In the first blog of this series, "Imposters at the Gate: Spotting Remote Employment Fraud Before It Crosses the Wire”, we armed you with tactics to spot potential signs of Remote Employment Fraud (REF) in the pre-hire stages of employment. We explored the goals of these threat actors, emphasized the critical importance of pre-hire detection and highlighted the need for security teams to partner closely with HR, talent acquisition, and hiring managers to spot early warning signs.
They made it through the interview process, checked all the HR boxes (yes, including their background check!)… and now they're on your network. What happens next? This blog pulls back the curtain on the technical indicators that expose REF actors during onboarding and before a full-blown breach. Get ready to deploy actionable Splunk detections, reclaim control of your onboarding pipeline, and shut down these threats before they do damage.
Before we get to the code, let's take a moment to address something important up front: it takes multiple indicators and behavior patterns to identify this threat type. For every example we’re about to share, there may also be legitimate reasons for this behavior. Looking at these examples in concert with other activities is critical, as is partnering closely with your HR and Legal teams. We strongly recommend taking a risk-based approach, comparing different indicators across both technical and non-technical platforms, before reaching a conclusion. Risk-Based Alerting (RBA), a feature of Splunk Enterprise Security, can be part of the solution. If you’re just getting started with RBA, The Splunk Guide to Risk Based Alerting is an incredible resource to learn how to get this set up in your Splunk environment. Leveraging RBA provides a more comprehensive view, ensuring a complete assessment that your HR and Legal teams will especially appreciate. Keeping this in mind, let’s explore the technical detection strategies.
Automated detection is unlikely before employees are hired, however the onboarding phase offers unique opportunities to detect suspicious activity. For example, the seemingly simple act of shipping IT assets, like a laptop, can reveal important clues.
The tech industry has widely reported that REF actors request shipments to addresses that don't match their stated location, often spinning elaborate (and hard-to-verify) reasons in the process. The actor will usually offer an explanation for such a request that is logistically difficult or could even infringe on privacy rights to corroborate (perhaps a medical emergency or a temporary relocation). Their reasoning for mismatched location and asset shipping addresses could be totally legitimate, but paired with other behaviors, there may be more to the story.
Identifying these inconsistencies isn’t always easy. Manual validation is possible but impractical in large organizations. Splunk can help to identify these inconsistencies by correlating data between applicant tracking systems and laptop shipments. Now is the time to sync up with your asset management team and learn more about the IT asset deployment and shipping process for new employees in order to get this data into your Splunk deployment. Additionally, you’ll need data on new joiners to your organization — this could be in an applicant tracking system or other repository of future employees.
The following query uses data from a ServiceNow instance joined to employee Workday information to identify when a potential employee requests a laptop to be shipped to a location which is inconsistent with their purported physical location.
index=servicenow sourcetype=laptop_shipment ```laptop shipment data``` | eval delivered_location=case(arrivalState="CA","California", arrivalState="TX","Texas", arrivalState="WA","Washington",arrivalState="VA","Virginia",arrivalState="OR","Oregon") ```specify the most commonly shipped to locations for string matching``` | table name Shipped_status arrivalTime employeeId Email home_state postalCode arrivalState delivered_location arrivalLocation | join type=outer Email ```compare laptop shipment data joining on employee's email with employee record data. If employee does not have email yet, then the employee ID could also be used to correlate``` [ search index=identity sourcetype=workday:employee earliest=-24h ```log source for employee record data``` | dedup name employeeId home_state] | search delivered_location=* | eval Suspicious=if(delivered_location==home_state,"No","Yes") ```if the laptop was delivered to a state that does not match their registered home address``` | search Suspicious="Yes"
The screenshot above illustrates an example of a user who purports to reside in Pennsylvania, yet the shipping address they’ve requested is in Oregon. A field in Splunk Enterprise was created called Suspicious to indicate when the user’s home_state field does not match the delivered_location field. This example could be easily expanded for global shipments and altered to meet your organizational needs.
REF actors will often use virtual private networks (VPNs) to conceal their true physical location. Threat actors mask their originating IP address and instead appear to be situated in any location where the VPN service has a node. This is one of several indicators detectable via technical means once the actor is in possession of a corporate device.
While VPNs hide REF actors’ real IP addresses, it also creates a new signal: traffic from VPN providers you don't expect. But what if you have remote employees who legitimately use VPNs? That's where baselining and exception-handling come in (more on that below). You can observe network traffic from suspicious VPN providers with ease in just a few steps using your identity provider (IdP) logs. Here is an example using Okta data and looking for commonly seen VPN/VPS providers.
index=okta sourcetype=OktaIM2:log ```enter your IdP log source``` debugContext.debugData.tunnels IN (*Astrill*,*Azire*,*CyberGhost*,*Express*VPN,*HideMe*,*IPVanish*,*Mullvad*,*Nord*VPN*,*OVPN*,*PIA*VPN*,*Proton*VPN*,*Pure*VPN*,*Slick*VPN*,*Surf*Easy*,*SurfShark*,*Star*VPN*,*TorGuard*,*TorProxy*,*Tiger*VPN*,*TunnelBear*,*Unblock*VPN*,*Warp*VPN*,*WarpSpeed*,*VPNReactor*,*VPN*Shield*,*VPN*Super*VPN*,*ZenMate*) ```listing of commonly used known VPN providers. Add or remove whatever is appropriate for your environment``` | eval user=coalesce('actor.alternateId',user), user=mvindex(split(user, "@"), 0) | rename targetUserAlternateId as user client.* as * request.* as * ipChain{}.* as * geographicalContext.* as * debugContext.* as * debugData.* as * | eval status=case(match(_raw, "FAILURE"), "failure", !match(_raw, "FAILURE"), "success") | stats count values(status) as status max(published) as UTC min(_time) as firsttime max(_time) as lasttime values(target_data) as target_data values(displayMessage) as displayMessage values(eventType) as eventType values(city) as city values(country) as country values(action) as action values(src_ip) as src_ip values(outcome.*) as * values(user) as user by tunnels,_time,host sourcetype index | fillnull value="N/A" | convert ctime(*ttime) ```| where isnotnull(user)``` ```uncomment this line if you only want to see events where an actual user was recorded in the IdP```
The above list of VPN providers is not an exhaustive list, and we recommend updating your list of known and used VPNs regularly. To find the most used list of VPNs, you can search against your logs.
Your Identity provider may also offer some assistance here! Many of the commonly used IdPs allow you to block traffic from known VPN providers that may be inconsistent with your organizational policy. Below are screenshots from the Okta and Duo console as examples of how and where to block VPN anonymizer traffic. In Okta, this blocking mechanism is defined as a Network Zone, and once configured, it can be used to deny VPN traffic. Similarly, in Duo, authorized networks can be configured for a similar effect.
Okta administrative interface showing Network Zone and blocking configuration.
Here is an example from the IdP, Duo, showing that anonymous network providers (VPN) can be denied access, or require second factor authentication in order to proceed.
Geolocation data can be inaccurate or easily spoofed by REF workers. REF actors sometimes slip up and reveal their true location, creating what we call 'improbable travel' scenarios — logins from opposite sides of the world within minutes. But here's the catch: this detection requires careful tuning. You don't want to flag your VP of Sales just because they hopped from New York to London for a meeting. Focus on newly onboarded employees who wouldn’t typically be traveling for business so early in their role.
We can go even further to observe inconsistent or improbable travel that reveals threat actors’ actual physical location. While this may not occur often, it’s important to look out for. There are numerous log sources that can detect an improbable geographic location for a user or traffic routed through unusual locations. Some primary log sources include: IdP logs (Okta, Duo, Azure, etc.), Firewall logs (Cisco, Palo Alto, Fortinet, Checkpoint, etc.), EDR logs (Crowdstrike, Cisco CSE, FireEye, SentinelOne, etc.), and Chat Platform logs (Zoom, Webex, Teams, Google Meet, etc.).
The following query uses the Authentication Data Model and Splunk Enterprise Security to determine improbable travel by approximating a “speed” and distance a user would have to be traveling to login/work from disparate geographic locations in the timeframe listed. It then looks at the results where speed and distance are greater than a threshold you can set. The query below is a comprehensive set of SPL. While you'll need to adapt it to fit your specific environment, the underlying calculations and logic are reusable with your own data. We encourage you to fine-tune your queries to run in your environment, including limiting the speed or distance threshold.
| tstats summariesonly=true values(Authentication.app) as app from datamodel=Authentication.Authentication where ((index="okta" AND sourcetype="OktaIM2:log") OR (index="firewall" AND sourcetype="pan:globalprotect")) ```enter your appropriate Auth log sources, and ensure Authentication Datamodel is being used and accelerated``` AND Authentication.action="success" AND Authentication.app IN ("Workday", "Slack", "*GlobalProtect", "Jira*", "Atlassian Cloud", "Zoom") AND NOT Authentication.user="unknown" by _time index sourcetype host Authentication.user Authentication.src span=1s ```pick what Apps are important to you to monitor geographic improbable access. These are some examples``` | `drop_dm_object_name("Authentication")` | fields user,src,app,_time,count,host | eval user=lower(replace(user, "((^.*\\\)|(@.*$))", "")) ```normalize how user appears``` | join type=outer user [| inputlookup identity_lookup_expanded where user_status=active ```Asset and Identities lookup from Splunk Enterprise Security that holds all identities and their respective information``` | rex field=email "^(?<user>[a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$" | rename email as user_email bunit as user_bunit priority as user_priority work_country as user_work_country work_city as user_work_city | fields user user_email user_bunit user_priority user_work_country user_work_city] | eventstats dc(src) as src_count by user | eventstats dc(user) as user_count by src | sort 0 + _time | iplocation src | lookup local=true asn_lookup_by_cidr ip as src OUTPUT ip asn description ```lookup that comes with Splunk that tracks ASN information``` | eval session_lat=if(isnull(src_lat), lat, src_lat), session_lon=if(isnull(src_long), lon, src_long), session_city=if(isnull(src_city), City, src_city), session_country=if(isnull(src_country), Country, src_country), session_region=if(isnull(src_region), Region, src_region) | eval session_city=if(isnull(session_city) OR match(session_city,"^\s+|^$"), null(), session_city), session_country=if(isnull(session_country) OR match(session_country,"^\s+|^$"), null(), session_country), session_region=if(isnull(session_region) OR match(session_region,"^\s+|^$"), null(), session_region) | where isnotnull(session_lat) and isnotnull(session_lon) | eval session_city=if(isnull(session_city),"-",session_city), session_country=if(isnull(session_country),"-",session_country), session_region=if(isnull(session_region),"-",session_region) | streamstats current=t window=2 earliest(session_region) as prev_region,earliest(session_lat) as prev_lat, earliest(session_lon) as prev_lon, earliest(session_city) as prev_city, earliest(session_country) as prev_country, earliest(_time) as prev_time, earliest(src) as prev_src, latest(user_bunit) as user_bunit, earliest(app) as prev_app values(user_work_country) as user_work_country by user | where (src!=prev_src) AND !(prev_city=session_city AND prev_country=session_country) AND ((isnotnull(prev_city) AND isnotnull(session_city)) OR prev_country!=session_country) ```looking for when the previous IP and current IP do not match, or the previous city and current city do not match, or previous country and current country do not match``` | `globedistance(session_lat,session_lon,prev_lat,prev_lon,"m")` ```built in command to help calculate distance``` | eval time_diff=if((_time-prev_time)==0, 1, _time - prev_time) | eval speed = round(distance*3600/time_diff,2) | eval distance= round(distance,2) | eval user_work_country=case(user_work_country="usa","United States", user_work_country="cze","Czechia",user_work_country="pol","Poland", user_work_country="ind","India", user_work_country="fra","France", user_work_country="can","Canada", user_work_country="mys","Malaysia", user_work_country="kor","South Korea", user_work_country="aus","Australia", user_work_country="bel","Belgium", user_work_country="dnk","Denmark", user_work_country="bra","Brazil", user_work_country="deu","Germany", user_work_country="jpn","Japan", user_work_country="che","Switzerland", user_work_country="swe","Sweden", user_work_country="zaf","South Africa", user_work_country="irl","Ireland", user_work_country="ita","Italy", user_work_country="nor","Norway", user_work_country="gbr","United Kingdom", user_work_country="hkg","Hong Kong", user_work_country="chn","China", user_work_country="esp","Spain", user_work_country="nld", "Netherlands", user_work_country="twn","Taiwan", user_work_country="est","Estonia", user_work_country="sgp","Singapore", user_work_country="are","United Arab Emirates", 1=1,"N/A") ```popular countries that appeared in IdP logs and normalized them to their fullname``` | lookup local=true asn_lookup_by_cidr ip as prev_src OUTPUT ip as prev_ip asn as prev_asn description as prev_description | eval suspect=if(!user_work_country==session_country,"Sketchy","Normal") ```determing if a user's work country matches their current country``` | search (speed>500 AND distance>750) ```this should be geographically improbable within the given time period``` | table _time,prev_time,user,host,src,prev_src,app,prev_app,distance,speed,suspect,session_city,session_region,session_country,prev_city,prev_region,prev_country,user_priority,user_work_*,prev_ip,ip,asn,prev_asn,prev_description,description | rename _time as event_time | convert ctime(event_time) timeformat="%Y-%m-%d %H:%M:%S" | convert ctime(prev_time) timeformat="%Y-%m-%d %H:%M:%S" | eval problem=if(!session_country==prev_country AND (!session_country==user_work_country),"Yes","Nope") | search NOT (prev_city="-" OR session_city="-") AND NOT [inputlookup known_devices_public_ip_filter.csv | fields ip | rename ip as src] ```add lookup of known IPs to remove False Positives``` | dedup user host prev_src src | fillnull value="N/A" | search problem="Yes"
REF actors have been identified using a combination of virtual audio and video devices in their operations to obfuscate their actual location on Cisco Webex / Zoom / Microsoft Teams calls. This technique makes it appear as though audio/video calls are originating from a location consistent with the employee or candidate, when in actuality, the threat actor is forwarding audio/video traffic from their true physical location, through an intermediate location (likely a laptop farm).
Detecting this activity requires careful analysis, regular review, and a thorough understanding of the audio and video devices commonly used within your environment. The key for detection here is to baseline your environment first to identify the outliers. (Think: a brand-new employee is suddenly using a virtual camera that no one else in the company uses.) Audit logs from teleconference and collaboration tools such as Zoom, Cisco Webex, and Microsoft Teams can also help determine devices used by REF actors. The search below can help identify non-standard or atypical video/audio devices used across these types of platforms.
index=zoom sourcetype=zoom:metrics:meetings:participant ```input appropriate audio/video log source``` camera=* microphone=* speaker=* ```looking for logs that contain information about camera, microphone, and speakers that are being used``` NOT (camera=*iPhone* OR camera="*FaceTime*" OR speaker="*AirPods*" OR camera="*MacBook*" OR microphone="*MacBook Pro Microphone*") ```Usually REF actors will use a combination of camera/microphone/speakers that are unique in the environment. Regardless if a Mac or Windows is used, there will typically be at least one odd indicator of tooling that is not commonly seen across the general employee user base``` ```use the rare command one at a time to scope cameras, microphones and speakers. Uncomment each line when needed below, and ensure only one rare command is uncommented at once``` | rare camera limit=50 ```search to show top 50 rarest cameras``` ```| rare speaker limit=50``` ```| rare microphone limit=50``` ```once you know what the rarest speakers, cameras and microphones are, you can specify them in the search directly```
Depending on which actor is currently managing the employee persona, a REF actor may use hop points to obfuscate their location or the location of their corporate device. This can be detected by carefully reviewing and correlating relevant logs. Correlating between multiple platforms’ logs, such as Zoom and Workday HR, is a standard best practice for identifying discrepancies. In the code example below, Zoom logs were used along with a user’s HR logs (from Workday).
index=zoom sourcetype=zoom:metrics:meetings:participant ```input appropriate audio/video streaming log source``` camera="*Stream*" (speaker="*Headphones*" OR speaker="*DarkStar*" OR microphone="*Tiny*") ```Declare any known rare combos of cameras, microphones, or speakers that are uncommon in your environment. These are examples``` | table join_time leave_time leave_reason speaker camera location microphone user_name role pc_name email ip_address | where isnotnull(role) ```only show results where a user is either an attendee or a host in a Zoom meeting``` | stats count min(join_time) as join_time max(leave_time) as leave_time values(pc_name) as pc_name values(leave_reason) as leave_reason by ip_address speaker camera location microphone user_name role email ```location is in logs but it's generic so we also use iplocation on ip_address``` | iplocation ip_address | rename email as work_email | join type=outer work_email [ search index=identity sourcetype=workday:employee earliest=-24h ```join to HR data, in this case data from Workday``` | spath All_Work_Addresses output=Workday_location | spath primaryWorkEmail output=work_email | spath Employee_ID output=employeeNumber | spath Preferred_Name output=name | spath "Location_Address_-_State__United_States_" output=state | table Workday_location name state work_email employeeNumber | dedup name employeeNumber state] | eval problem=if(Region==state,"No","Yes") ```Region comes from iplocation on the IP address in Zoom and state comes from Workday logs``` | search problem="Yes" Workday_location=* ```Find results where there is no match``` | fields - lat lon | table ip_address speaker camera microphone location City Country Region state Workday_location problem ```work_email hidden from results for privacy reasons```
As shown in the example search results below, Zoom logs state that the REF workers are coming from California and Virginia. However, in Workday, the employees are listed as being based out of either Texas, Maryland or Idaho. The Idaho HR location never matches up to the Zoom logs coming out of Idaho. Instead, for that potential REF worker, it shows them in California, and not Idaho. The field “problem” was created to indicate that there is a geographic mismatch between an employee’s Zoom device location and their registered HR location.
Example events demonstrating inconsistent worker locations
Looking more deeply at network traffic, particularly from video conferencing applications, can be useful to corroborate other evidence. In some cases, particularly high latency has been observed from these threat actors — well above what’s normal for the majority of employees. While latency could simply indicate a slow network connection, when combined with other indicators, it can help build a more complete picture. Typically, REF workers use various forms of streaming software on their computers, from a remote location and often connecting to laptop farms, which can result in latency, low frame rates, or very low resolution settings. Often, the laptop farms that these REF workers use will have various intermediaries or change their tactics, which makes detection easier. In the SPL below, a baseline was taken of the environment and determined that 150ms of video latency was quite high and unusual, increasing this to 300ms highlights the outliers even more.
index=zoom ```use appropriate audio/video data source``` | spath "payload.object.participant.qos{}.type" | search "payload.object.participant.qos{}.type"=video_input ```focus on video input, as that's what would register lag in the most obvious manner``` | rename payload.object.participant.qos{}.details.avg_latency as avg_latency "payload.object.participant.qos{}.details.latency" as latency payload.object.participant.email as email | rex field=avg_latency "(?<average_latency>\d+) ms" | rex field=latency "(?<overall_latency>\d+) ms" | search email="*@splunk.com" ```put in correct company email domain``` | table email overall_latency latency avg_latency average_latency _raw | stats latest(overall_latency) as overall_latency by email _raw | where overall_latency>300 ```150ms of latency should be very noticable to end user```
Remote access tools provide multi-functional abilities to threat actors. These tools share similar functionalities—file transfer, remote control of a system, and device redirection, for example. An REF actor may use these functionalities to complete a variety of objectives in-network, including exfiltrating files, installing malware, and remotely working on the system to obfuscate location. Use of this software can similarly be detected. Certain corporations may have licenses for remote access tools like Splashtop or TeamViewer, where they are using the paid version and the company is sponsoring it. Paid tools with licenses are less commonly seen in REF operations. It is important to baseline an environment to determine which remote access tooling or programs may be allowed, and what is prohibited. The best way to interdict this behavior is to block the application from ever being executed on a workstation, utilizing something like Jamf/JumpCloud/Intune/AppLocker, or even EDRs like Crowdstrike/SentinelOne/Cisco CSE, etc.
Using readily available lists of remote access tools, you can search endpoint logs for unauthorized usage. Determine what tools are prohibited or allowed in your environment, and then run queries against your EDR data to see if anyone has utilized those tools. The following are examples of GitHub repositories containing names of this style of app.
Heads up: These repos are great resources for enumerating remote access tools, but they will go stale over time. Always double-check the source and do not blindly trust it as a cure-all.
index=edr* sourcetype=crowdstrike:falcon:fdr:* CommandLine ```input appropriate EDR log source that contains CommandLine and Process Application information``` IN ( *Splashtop*, *LogMeIn*) ```these are just examples. Recommendation is to pull a full list of RMM tools from various Git Repos and input the names inside of the IN statement``` OR AppName IN (*Splashtop*, *LogMeIn*) ```input the same examples above here``` ```exclude any known processes or Apps``` | eval App=coalesce(AppName,ParentBaseFileName) ```combine the appname and parent file name if known``` | fillnull value="null" | stats values(App) as App by AppName event_simpleName ParentBaseFileName ImageFileName CommandLine ```aggregate on eventtype, Commandline, process, parent process, and App name``` | convert ctime(*time) | sort 0 + f_time | stats count values(ParentBaseFileName) as ParentBaseFileName values(CommandLine) as CommandLine by event_simpleName App | table App event_simpleName ParentBaseFileName CommandLine
It’s crucial to reiterate that, none of the strategies above, when viewed in isolation, can confirm the existence of REF within your organization. Rather, multiple indicators should be evaluated together to create a higher confidence assessment.
RBA is great for this use case. It enables your organization to aggregate indicators and observe behavior patterns that may be inconsistent with typical remote employees. Taking it further, these individual cases can be scrutinized more carefully; non-technical indicators can be brought in, and a more holistic assessment can be made in coordination with HR and legal teams in your organization.
You now have some powerful Splunk detections to help identify REF actors during onboarding and beyond. While attacker techniques will always evolve (it’s the nature of the game), having detections in place is a critical first step, and knowing what to look for is half the battle
The next challenge — the response — is where things get complex, especially when navigating the intricate landscape of HR & legal. Unlike traditional technical threats, responding to REF requires a level of close partnership atypical for many tech and HR teams. In our final blog, we’ll pull back the curtain on how we partnered across Splunk to build a process for identifying, researching, evaluating, and escalating REF cases; a step-by-step guide to building a cross-functional response plan that works in the real world. You won't want to miss it.
The world’s leading organizations rely on Splunk, a Cisco company, to continuously strengthen digital resilience with our unified security and observability platform, powered by industry-leading AI.
Our customers trust Splunk’s award-winning security and observability solutions to secure and improve the reliability of their complex digital environments, at any scale.