**Product Tests

What is [[TEST]] all about?!?!

Macro: Site Tests::User (part of test group) asking what is test all about?

Thanks for reaching out. We frequently try out new features -- both big and small -- by conducting testing with a small group of users. This is the standard practice for most companies, and tests can sometimes last weeks or just a few days. In order to not impact or skew test results, we typically try not to share details about tests while they are ongoing. While we can’t address your questions about this test, we do want and appreciate your feedback. We will be glad to pass your comments on as we work to determine which potential features our customers like and don’t like. Thanks!

Why can’t you tell me more about this test?

Macro: Site Tests::User (NOT part of test group) asking what is test all about?

Thanks for your question. When we launch a test, whether it’s for a day or two or several weeks, we are most interested in feedback from the users who are part of the test group. These customers are experiencing the testing of a new feature firsthand and we want their honest, objective feedback. When we share details about a test there is always the concern that sharing too much will affect a user’s natural response to testing and invalidate their feedback. For that reason, we will in most cases not discuss the specifics about a test. While we can’t address your questions, we do want and appreciate your feedback. In fact, our design team and engineers are counting on it. We will be glad to pass your comments on as we work to determine which potential features our customers like and don’t like. Thanks!

So you aren’t going to tell us about new features!?!?

Macro: Site Tests::User upset about lack of info

Thanks for your question. We will share news about new features just as we have done in the past. However, while we are testing out possible new features (and their variations) we typically will not share our test plan or specific details. When we share details about a test there is always the concern that sharing too much will affect a user’s natural response to testing and invalidate their feedback.

This was launched on August 28, 2023.

• What customer problem is this solving: As a user, I get frustrated and lose my trust in Upwork when I see irrelevant results when searching.

• Hypothesis: If we can better understand queries by segmenting terms that are meant to represent 1 concept, we will have higher precision in our search results which will lead to an increase in successful sessions.

• New experience: Updated query segmentation strategy which includes an updated match strategy. This is an algorithmic change so it’s a backend update.

*UFS: User freelancer search--refers to the system by which registered clients search for freelancers.

Tentative launch date: September 19, 2023

Problem:

Episodic use of Freelancer Plus is common as 34% subscribers have canceled then re-enrolled at least once and 7% report that cancellation is expected to be temporary. However, Freelancers face a 2-step process to pause a subscription;  requiring cancellation, then reactivation. Friction in this process contributes to our leaky bucket, where average retention on Freelancer Plus is just 17.3% at 12 months. Conversion rate also presents an opportunity at just 7.4% at the Membership Plan comparison page.

Hypothesis/Solution:

By introducing a feature that enables Talent to skip a month, we reduce friction in resuming a subscription after a pause. We hypothesize less friction will improve retention. We also hypothesize that clear signals of flexibility will lead to more conversions to Freelancer Plus

“Skip next month” option  in Membership & Connects page

“Instead of switching, do you want to skip next month?” pop up in Membership plans page 

Resources:

These Help Center articles will have customer-friendly details and the FAQs below about this test once it's launched (tentative launch date 9/19/2023):

• Freelancer Plus
• Membership Billing

Below are the FAQs that will be added to the abovementioned Help Center articles:

Why would I use the “Skip Next Month” feature rather than canceling my subscription to Freelancer Plus, then re-enrolling?

We built the Skip Next Month feature to make it more convenient for you to access the Freelancer Plus features exactly when you need them. You no longer need to re-enroll in Freelancer Plus when you know you’re taking a break from the platform. You can simply skip a billing cycle, after which you will be automatically re-enrolled.

Will my Plus membership automatically resume the following month if I skip a month of Freelancer Plus instead of canceling?

The short answer is yes. The “Skip Next Month” feature allows you to downgrade to Freelancer Basic for the next billing cycle without needing to re-enroll in Freelancer Plus. You will be automatically re-enrolled in Freelancer Plus in the following billing cycle.

Why don’t I have the option to skip a month of Freelancer Plus instead of canceling? Can I opt-in?

We’re currently testing this feature so it’s not available to everyone yet. There is not an opt-in for this test.

Can I choose to skip multiple months?

You can only skip the next billing cycle (i.e., you can skip one month at a time). Should you want to skip multiple months you would need to select this feature again before you are automatically re-enrolled in Freelancer Plus. There is no limit to how many times you can use this feature. 

This was launched last September 4, 2023.

Problem:

It’s not easy for new CLs to find talent on UW today that is relevant and hence hard to take an action to move forward. There is too much friction today to find the most relevant talent. “I need to browse through pages of irrelevant search results or marketing content before I see relevant talent that can do my work.”

Hypothesis/Solution:

We believe reducing friction and improving talent relevance for clients by showcasing top-rated talent with proven experience within (3 clicks or less) will help drive first time client seeking work (CSW) through higher client interaction much more than our current user experience.

Also, we believe by introducing a Direct Message feature in this experiment, we will match CLs mental model better of being able to “browse and buy” and CLs will be able to start a conversation and start working with talent directly, and we will not only improve new CSW but contract starts as well.

Details

• Allows non-registered CLs (50% QT) to find TR Talent relevant to what they’re looking for and message them directly

• Today - most CLs don’t know what to do when they get to upwork.com, so they click “Get Started” where we force CLs to register right away before even seeing Talent we know most drop off at this point because they haven’t gotten to see if we have anything relevant to offer them

• New:
  • CL has more of a self-service option to filter through Talent based on exactly what they’re looking for
  • They can set up a DM to send them, BUT before it sends…
    • They’re forced in email / phone verification, THEN have the ability to send their message
    • They can send message to that same FL they chose, of if they chose as “unavailable”, they can click to send to other related FLs to increase chances of hire

• While CLs will not be a part of the normal eligibility requirements we share here anyone in this specific cohort will have their own eligibility requirements:
  • CL must bypass phone AND email verification to send a direct message to FL
  • CL can only send messages up to 5 new FLs per day
  • CL will still be able to send messages (up to 5 FLs a day) if they log out and then return to their account
  • CL cannot send additional messages to FL until the FL responds back

Screenshots

Tentative launch date for this was on August 31, 2023.

We will improve the relevance of MFS (merchandising freelancer search) by expanding the 1st pass recall. This will help relevance by adding FL profiles that have Semantic Text Similarity (STS) with the job post description. This is a back-end change.

Tentative launch date for this was on August 31, 2023.

In FWH Best Match feed, a legacy model is used to predict the probability of FL hired by the job and jobs are ranked by a linear combination (logistic regression) of lightfm embedding similarity, hire probability and other features. We trained a new deep learning model that outperforms the production model in offline evaluation. We believe the new model is able to better recommend relevant jobs to FLs and increase FL engagements in Best Match feed.

Test Cells

Find Work Home (Freelancer Landing page when they are logged in and come to Upwork)

This was launched on September 15, 2023.

Problem we’re solving: Best Match algorithm suggests irrelevant jobs. By using a combination of two models we can improve the recall set of freelancers, thus improving the overall feed relevance.

This QT is to test the relevance of a modernized algorithm for FWH Best Match feed. We will modify the 1st pass so it uses 1) STS: for semantic similarity of jobs-FL_profiles; and 2) Two Towers: for location category matching.

This was launched on September 22, 2023.

• Problem Statement
  • VTB Collaboration is a set of feature updates to the VTB functionality in order to bring it to a state that it can enable the transition of Enterprise Clients from Talent Clouds to VTB as a talent management solution.
    With VTB Collaboration V1 we introduced a barebone sharing capability that allowed multiple hiring managers to work together on a single VTB list and make changes to it based on specific permissions. Since the V1 was just covering the sharing aspect, the hiring managers, who have been sharing their lists and lists have been shared to, lack visibility into the list changes that have been made.
• Hypothesis
  • By providing visibility into the list changes that have been done by other hiring managers we will bring the Enterprise team one step closer to sunsetting the Talent Clouds functionality.

From September 21, the launch date for this was moved to today, September 25, 2023.

Problem:

As Upwork, we urgently need to identify a viable replacement for Grammarly’s SDK, (which is currently integrated on Catalog, Cover Letters and Messages) in light of its January 2024 scheduled deprecation. Grammarly has proven to enhance FLs’ experience and contribute to the business as reflected in a 4.4% increase in FLE in a prior QT (Grammarly proposal letter QT). 

Hypothesis/Solution:

If we replace Grammarly with a competitive vendor, then we will maintain/increase FLE (along with other contract outcomes success metrics like contract cancellations and client private feedback) because FLs will have access to a competitive, if not better, text editing service. 

We currently use that integrated in Catalog and cover letters, & it's being tested in Messages. As it's proven useful for FLs, this test is essentially testing a possible replacement for Grammarly Text Editor in these areas to see if it's just as useful to FLs.

This was launched on October 4, 2023.

Problem we’re solving: 

On April 26, Shamrock went live and became the new status quo. All existing clients and new clients migrated to $2.95, which is the default value in Shamrock. However, there will be an exception: two small groups of existing clients were experimented on with contract initiation fee amounts different from the $2.95 baseline. These two groups will have a $0 and a $4.95 contract initiation fee, respectively.

Currently, the experiment result for existing clients is favoring the $4.95 contract initiation fee. However, we cannot default all clients to $4.95 without conducting a new client test. This is because new clients may react differently from existing clients when faced with higher contract initiation fee. Most importantly, we want to check if the start rate is significantly different for new clients. 

Hypothesis/Solution:  

We would expect New Clients to respond to a $4.95 CIF similar to Existing Clients; Some additional consolidation of Contracts but no impact on FLE Spend.

Customer-Facing Comms

For the $4.95 QT for new clients,  we will not notify new clients they are in the test (i.e. no in-product notifications or announcements). We are preparing our customer-facing teams with reactive FAQs.

External FAQs (Reactive) 

Reactive FAQs for answering customers’ questions - Reactive FAQs are not published/not public-facing, only used if agents get these specific questions and need answers:

Reactive freelancer-facing FAQs

What is the Contract Initiation Fee?

• Clients will pay this initiation fee on Talent Marketplace and Project Catalog contracts. This is a once-per-contract fee when clients make their first payment to a freelancer.  This fee is included in our agreement page.

When will this fee be collected?

• The exact time when the fee is collected depends on whether it is a fixed price or hourly contract.  

• Hourly contracts: Collect fee when clients make their first payment to freelancers
• Fixed-price contracts: Collect fee when clients fund the first milestone 

• This fee will be collected for each contract, but only once-per-contract.

I already pay a Client Marketplace Fee. Why is there a separate Contract Initiation Fee?

• Our aim is to foster long-term relationships that allow you to identify and work with freelancers most qualified to perform the work. This fee reflects the value in creating an effective match between freelancers and clients for a given scope of work. Learn more about making payments for our ​​Client Marketplace and Contract Initiation Fees.

Why do I have to pay this fee? 

• The fees on our platform cover infrastructure maintenance, platform investments and feature improvement costs.  We assess this fee when you have selected a professional to complete your project and made your first freelancer payment for each contract.

On a fixed-price contract, do I have to pay the contract initiation fee when I fund the contract or first milestone?

• You will pay the once-per-contract initiation fee at the same time that you fund the first escrow deposit for fixed-price contracts. However, if the contract is canceled and no payment is released to the freelancer, we will return the contract initiation fee. 

Do you charge the contract initiation fee again if I add new milestones to an existing contract?

• No, if you are continuing to work on an existing contract and just adding milestones, you will not have to pay the contract initiation fee again.

Do I have to pay the contract initiation fee when I purchase a project or schedule a consultation? What happens if the project or consultation is canceled?

• Yes, you will pay the fee when you purchase a project or consultation.. However, if the project or consultation is canceled with no payment released to the freelancer, we will return the contract initiation fee to you as well as your escrow deposit. 

Does the fee amount vary by contract value, and how will I know how much my fee will be?

• No, the contract initiation fee is a flat fee of up to $4.95. You’ll be able to see the exact amount for your contract when you check out. The contract initiation fee shows as a separate line item.

If a freelancer refunds my contract payment, will you also refund my contract initiation fee?

• No, the once-per-contract initiation fee will not be refunded after you have made a payment to a freelancer, even if the freelancer issues you a refund. 

If I dispute a contract and win my dispute, will you refund my contract initiation fee?

• No, the once-per-contract initiation fee isn’t eligible for a refund after you have made a payment to a freelancer, even if the contract payment to the freelancer is refunded due to a dispute. 

Where can I see the contract initiation fee in my invoices and reports? Is it included in the Weekly Billing Summary?

• You’ll have a separate invoice for the contract initiation fee, which includes the fee and any applicable taxes on the fee. You can also see the fee and any applicable taxes as line items on your Transaction History Report. It does not display on your Weekly Billing Summary, which reflects payments to freelancers.

Will I pay the contract initiation fee if I hire an agency?

• Yes, you will pay the contract initiation fee for the start of a new contract with an agency.

If I hire multiple freelancers for one job through an agency or through a team proposal, will I have to pay the one-time contract initiation fee on each freelancer’s contract?

• Yes, each new contract with a different freelancer will have a once-per-contract initiation fee, even if all the freelancers work for the same agency. If you hire 3 freelancers to work on a job, you’ll pay the fee 3 times.

If I rehire a freelancer I’ve worked with before, will you charge the contract initiation fee?

• Yes, the fee is per contract. If you start a new contract, you will pay the contract initiation fee even if you have worked with the freelancer before.

If I close a contract and create a new one to switch from fixed-rate to hourly, or vice-versa, with the same freelancer, will I pay another contract initiation fee?

That depends on whether you have already activated the contract with a payment to the freelancer. Here’s how it works for hourly and fixed-price contracts:

• For contracts that started as hourly: 
  • If the freelancer hasn’t logged any hours yet, you can cancel without a contract initiation fee on the initial contract, since we don’t charge the fee until you make your first hourly payment. 
  • If the freelancer has already logged hours, you will pay the contract initiation fee on both contracts. You will need to pay the freelancer for the hours they already logged, and we will charge this fee. Then you will pay another contract initiation fee when you fund the first milestone of the fixed-price contract.
• For contracts that started as fixed-price: 
  • If you have already funded a milestone or purchased a project but have not yet released a payment to the freelancer, you can get a full refund as long as your freelancer agrees. 
  • If you’ve already released at least one payment to the freelancer, you will have to pay a contract initiation fee on both contracts.

Are you charging the contract initiation fee on Enterprise, Upwork Payroll, BYO, Any Hire, Direct Contracts, or Talent Scout contracts?

• No, those services have their own fee structures designed specifically for the features they offer. We only charge this contract initiation fee on Upwork Marketplace and Project Catalog contracts.

Will the client marketplace fee apply to the contract initiation fee?

• No, we do not apply an additional marketplace fee to the contract initiation fee. The client marketplace fee only applies to the contract value for the amount you pay the freelancer.

Internal FAQs 

When will the $4.95 QT for New Clients launch?

• October 4, 2023.  We expect the test to run for approximately 31 days to gather the required data. While we don’t foresee the test to run longer than that, we do sometimes adjust the length of the test depending on early insights. 

Is the Contract Initiation Fee new?

• No, the Contract Initiation Fee was introduced to both existing and new clients on April 26, 2023 as part of Project Shamrock.  Existing clients and new clients migrated to a default $2.95 Contract Initiation Fee, with small groups for testing varying fee amounts.  Clients pay this initiation fee on Talent Marketplace and Project Catalog contracts.  

What is the purpose of this new client test?

• This test will inform a decision on whether we should apply a $4.95 fee to all clients. We cannot default all clients to a $4.95 fee without expanding our test to New Clients.

Do clients know they are included in the test?

• Clients will not be notified that they are part of a test. New clients will not have previously been exposed to different Contract Initiation fee amounts.

Who is paying this fee?

• We will assign new clients to the follow group::
  • Control: Status quo $2.95 contract initiation fee (50% Allocation)
  • Treatment: $4.95 contract initiation fee. (50% Allocation)

Who is included in this test?

• This test addendum applies to new Talent Marketplace Clients. Enterprise clients are excluded from this test.

How do you determine who is a new client?

• We consider a new client to be a client who has not made their first job post. When clients make their first job post, they will be eligible for this test.

Why are we testing a $4.95 fee with new clients only?

• We are expanding the current test to include New Clients. The $4.95 test cell was only applied to existing clients so far.

How much are existing clients paying compared to new clients?

• Existing clients have been part of a test with varying fee amounts of $0, $2.95 and $4.95.  By extending this test to New Clients, we will be treating New and Existing Clients the same (except there will not be a $0 cell for New Clients).