(Internal) Pipeline Analytics Debugging

Audience

Internal

Displayed Description

Page Type

Article

Product	Pipeline Analytics (Talent Compass)
Expert(s)	Shalan Billault-lee, Alex Shadley (CRM team)
Slack channel
This article was last verified on	04/25/2024

🔍 Articles in This Section

Please use the following list to see additional internal articles regarding Pipeline Analytics:

(Internal) Pipeline Analytics Overview
(Internal) Pipeline Analytics Filter Guide
(Internal) Pipeline Analytics Debugging (📍you are here)

How we currently debug Pipeline Analytics questions

Most Common QuestionWe Get

“These people are missing from pipeline analytics. They should be in this stage, but I don’t see them”
They fall into two buckets:

There is an actual bug

Ultimately, we were missing scorecard information in Pipeline Analytics
example of an actual bug

Missing data is because customer does not understand how our PA logic and filters work

Candidates not sent an outreach or logged touchpoint through Gem won’t show up in Gem Candidates
Candidate won’t show up in a stage unless we have a scorecard, or the interview date has passed
example of more education needed (over the course of 24 days)

Steps we are already taking to debug

When a pipeline analytics question comes in through support, these are the first few steps I take

Check for: who I’m talking to (recruiter, sourcer, HM)
Look at report link they’ve provided (if they provide one, otherwise i’ll ask)
look at application link they provided (if they provided one)
run application link through debugger
if Gem user, look at Gem activity

The first set of steps goes fairly smoothly, and quickly. For the next set, this takes up quite a bit of time and resources

Try to find the prospect in Pipeline Analytics based on debugger info
If not enough info exists, use numeracy queries to get:

stage
who gets credit
job
look at profile, to see if there are any other applications
application information

Consider stage mapping, or any other team specific nuances that may affect this (will their workflows affect things like Gem attributes, or outreach stats?)
File a bug. Example for Lyft (https://app.asana.com/0/1130275165571676/1166331035810649)

Gaps in Tooling for Troubleshooting

Easier ways to see if this is really a bug
Gaps in knowledge based on how Greenhouse and Pipeline Analytics logic works

Other Considerations

Examples of nuances in Greenhouse and Pipeline Analytic logic

If a candidate has multiple applications, we won’t attribute the second application unless the second application was opened before we closed the first. [from Slack Candidate Discrepancies, Cell E9)
If a candidate is hired, they become a private candidate. They count towards stats, but their name doesn’t show up in the list of candidates counting towards that stat. [from TripActions Troubleshooting, page 2-4]

Existing Resources

+Gem Stats, and What They Mean

Webhooks processing

Pipeline Analytics debugger (in product)

Numeracy Dashboard to troubleshoot Permissions (TripActionsGreenhouse Candidate Applications)

How Customer Success and Support Handle Education

Currently : CSMs and Support have (multiple) 1on1 sessions with individual users, RecruitingOps and Recruiting Leads to explain our Pipeline Analytics logic

Consolidating and responding to each individual case: Slack, Brex

In progress :

Creating and consolidating a “How to Onboard Customers onto Pipeline Analytics” Playbook + Training best practices
Focus on helping customer build trust and understand the data early on during the Onboarding phase

Cost of these issues

Approximately half of our 50-60 PA customers have run into this issue ( ~25-30 customers)
Estimated CS cost is ~3 hours per customer (?) = ~90 hours
Estimated eng cost is ~3 hours per customer (?) = ~90 hours
Some loss of customer trust, increased customer confusion
Adam spends 25% of his time handling Pipeline Analytics cases; Jess in the last 3 weeks has spent 1-3 hours daily debugging and answering customer questions.
The increase in cases correlates with CSMs prioritizing goal of “Driving Value with Reporting” by focusing on CRM training and “Making Gem really sticky” by expanding to other personas like Recruiting Leadership and Hiring Managers

Wishlist

Ideal: From a Gem candidate profile, see where this person is being counted in Pipeline

Would also help Team Admins add Manual Attribution to a candidate profile

Given a GH application link, short cut to where they will show up in Pipeline Analytics (or exhaustive list of attributes. This can even be a raw dump)

Nice to Haves

Tool tips over certain parts of Pipeline Analytics and / or Outreach Stats describing how we calculate certain values.
In product explanations for differences between “Gem Candidates” and “All Candidates”
More information surfaced in the debugger (Einas mentioned standard queries’ information could be simply surfaced in the product debugger)

TODO Ideas

Search over GH Candidates + Gem People by name in debugger (instead of pasting in GH links - though we should still support that too)

figure out best way to handle permissions here - should let ppl know when they can’t see someone?

Pull up all GH Apps, the Gem Person record (if it exists), and all send/reply/GH activity in one page in debugger (instead of just the one application)
Show list of groupable / filterable attributes in the debugger for the Person+GH Candidate - so all fields from the Person/Candidate/App/Job/etc that can be used in Pipeline Analytics, to make it clear where this candidate will show up. Could show these as “tokens” similar to how the new filters UI looks
Show all applicable permissions on the candidate + apps, so it’s clearer why a teammate might not be able to see someone that you can see
Show more info about the application, like close/reject timestamp, reachout window timestamp
Maybe a more compact view for when each stage transition occurred for the app, instead of just the big list of events from the current debugger
link to this “debugger” search from the “click stat” popup with “Not seeing a candidate you expect to see? Click here” - could take you to debugger with the context of which view you expected the candidate to show up in

Questions

better name for debugger? best places to link to it - where are users seeing missing data?
what tooltips should we add and what should they say
how to best convey Gem Cands / All Cands? It’s actually complicated. Ben’s new view makes it a toggle, but this still may not help much.

I made some quick slides to help with this. Our Gem Cands / All Cands names are actually pretty terrible: https://docs.google.com/presentation/d/1dyMXFvZpnMUPNhxM_awLNnQ7PQ0pQGt_yqLhQYCxuSY/edit#slide=id.g717e3b19a7_0_15
Maybe we should call it “All GH Candidates” and “People with Gem Outreach” or something…

Thoughts on keeping track of High-Priority PA Bugs(5.27.20)

Proposed Bug Escalation Process

CSM posts in #pipeline-analytics-support (potentially create Asana task? @Adam T to follow up with appropriate people), assign p1/2/3 based on definitions below, and tag PA Eng-on call
PA Eng-oncall resolves/assigns POC to resolve issue
PA Eng-oncall updates thread with estimated resolution time
PA Eng-oncall updates Asana thread when diff is landed

What constitutes a High-Priority bug?

Severity: Flagrantly wrong and visible data
Number of Users Impacted: All? Users

AND

Key Customers: Some ARR Threshold, coupled with who at the customer is asking (e.g. Recruiter vs. VP of Talent)

Comms

High-level Overview of System

3 point scale for bugs (0-3)
0 lowest pri, 3 highest pri
the idea is that a sliding scale will help CSMs more quickly identify how to appropriately handle bugs/escalate to EPD as necessary.
There will be 3 factors baked into the scoring system (Severity, Reach/Number of Users Impacted, ARR/Key Customer). 1 point for each of these that the bug reaches, so that we are not overindexing on an issue that a single large customer like Slack has that is low severity.

3 factors

Severity. Flagrantly Wrong and Visible data. Example, Funnel view shows 4 offers → 2 hires and pass-through rate is 100%.
Number of Users/Customers Impacted. Is this issue an isolated incident (a candidate missing on a single job for a single customer? Or is this systemic and happening for all customers?
Key Customer — Some ARR Threshold, coupled with who the main stakeholders the customer are, and additional customer context. Still trying to define this threshold but different issues carry different weight when it’s a large vs. small customer AND the stakeholder at the customer is a key DM. For example, Chauncey @ Slack reaching out vs. a sourcer carry different weights. This is the one factor that may be a bit subjective depending on outside context ( customer’s renewal coming up, etc.)

If a bug is defined as P0(score= 2/3), here’s a sample escalation process

CSM posts in #pipeline-analytics-support (potentially create Asana task? @Adam to follow up with appropriate people), assign p1/2/3 based on definitions above, and tag prospective EPD owner.
EPD Owner resolves/assigns POC to resolve issue.
EPD Owner updates thread with estimated resolution time and any pertinent details so that CS can give timely updates to customers.
EPD Owner updates Asana thread when diff is landed.

Examples of each type of bug

Bug Points	Examples	Example Scoring and Reasoning	What should I do in this case?
0 → Low Priority	Recruiter at a small (> 10k) customer reports a candidate missing in a single job. No other reports at company or from other users. (0 points) Severity → 0 point Number of users impacted → 0 point Key Customer → 0 point Sourcer at a mid-size (10k-25k) company reports that a candidate isn’t sitting in the Offer stage in Activity view. Severity → 0 points Number of users impacted → 0 points Key Customer → 0 points	Severity = 0 points. All signs point to these being an isolated issues and more often than not, candidates missing in a req could be user error/misunderstanding of how Pipeline Analytics works, thus 0 points. Number of Users/Customers Impacted = 0 points. All signs point to these being related to one users data vs an entire team or Gem wide; thus worth 0 points. Key Customer = 0 points. With a few exceptions, we generally wouldn’t put a sub-25k customer in the key customer bucket, thus 0 points.	Follow the standard bug reporting process and email eng-oncall to look into it when they have time. No need to post in #pipeline-analytics-support.
1 → Medium Priority	VP of Talent at a medium size company reports that a candidate isn’t sitting in the Offer stage in PA Activity view. Severity → 0 points Number of users impacted → 0 points Key Customer → 1 points Expected Hires calculator not pulling in proper pass-thru rates. Severity → 0 points Number of users impacted → 1 point Key Customer → 0 points	Severity. All signs point to these two examples being relatively mild issues, thus 0 points. Number of Users/Customers Impacted . We give the Expected hires calculator issue 1 point here because it impacts all users on Gem PA. Key Customer. We give the VP of TA issue 1 point here because the VP of talent at this company is the main stakeholder for keeping Gem.	Follow the standard bug reporting process and email eng-oncall to look into it when they have time. No need to post in #pipeline-analytics-support.
2 → High Priority	VP of TA at Lyft reports that dashboard isn’t some members of the Lyft team. Severity → 1 point Number of users impacted → 0 points Key Customer → 1 point Greenhouse Scorecard ID issue before Slack reported to us. Severity → 1 point Number of users impacted → 1 point Key Customer → 0 points	In both of these cases, we give two points, constituting high but not highest priority. Severity. We give 1 point to the both examples in this case because PA is non-performant, and showing flagrantly wrong data (candidates showing in stages they shouldn’t be), Number of Users/Customers Impacted . We give a point to the Greenhouse scorecard issue because it impacts most customer’s Greenhouse data. Key Customer. We give the Lyft example 1 point here because they they are one of our largest customers + VP of TA is main DM @ Lyft.	In this case, post in the #pipeline-analytics-support channel, create an Asana task, and gather the correct information from the customer(s) so that EPD can properly troubleshoot and fix.
3 → Highest Priority	Pipeline Analytics down for all or most teams at Gem. Severity → 1 point Number of users impacted → 1 point Key Customer → 1 point Large company (50k+) reports that all passthrough rates are greatly skewed because Gem is counting future activity in funnel view. Severity → 1 point Number of users impacted → 1 point Key Customer → 1 point Greenhouse Scorecard ID issue after Slack reported to us. See slack thread. Severity → 1 point Number of users impacted → 1 point Key Customer → 1 point	In all of these cases, the example bugs get 3 points because the dashboard is non-performant, affecting a critical mass of users, as well as a key or multiple key customers. Severity = 1 point. Number of Users/Customers Impacted = 1 point. Key Customer = 1 point.	In this case, post in the #pipeline-analytics-support channel, create an Asana task, and gather the correct information from the customer(s) so that EPD can properly troubleshoot and fix.

How we currently debug Pipeline Analytics questions
Most Common QuestionWe Get
Steps we are already taking to debug
Gaps in Tooling for Troubleshooting
Other Considerations
Examples of nuances in Greenhouse and Pipeline Analytic logic
Existing Resources
How Customer Success and Support Handle Education
Cost of these issues
Wishlist
Nice to Haves
TODO Ideas
Questions
Thoughts on keeping track of High-Priority PA Bugs(5.27.20)
Proposed Bug Escalation Process
What constitutes a High-Priority bug?
Comms
High-level Overview of System
If a bug is defined as P0(score= 2/3), here’s a sample escalation process
Examples of each type of bug