Speed up proof & price addition

[raphodn]

Currently

@serpico (extracted from #523)

In the same spirit/related, I was adding prices yesterday based on price tag's pictures I took the same day, for reference I timed myself doing it, for 16 prices it took me 54'( with a some back and forth between OFF and Open Price webapp to create products which didn't exist, but I believe that's not the bulk of the time spent ) so it took 3'40" per price, although I'm willing to admit my workflow could be enhanced, I do use a text file from which I copy/paste the EAN for the products I track, saving some time I think.
I noticed that for a session like that, the type of proof, the location, the date and currency would be the same and the user would save time if those were once entered at the beginning of the so-called "session" and "locked", some kind of streamlined interface without all those field after initially being entered, ideally making use of the keyboard's Tab key to jump from one field to the next...maybe that's mostly what we have for the receipt mode. An additional "lockable" parameter would be EAN as opposed to category - ideally jumping straight the the EAN input box to paste the relevant EAN.
After checking what the receipt mode UI looks like it would be a variant of it, if it's deemed worth exploring I'll expand in a specific issue and share screenshot of my vision of the UI, based on the current receipt mode.
Now the question is how many user engage ( percentage of prices coming from this opposed to receipt ) in this, how much time do we save ( here's I'm thinking a stopwatch feature could be useful although it might scare away user when they realize how much time they spend for so few prices - so maybe make it bonus interface ) and is it worth the time coding this specific interface.

Related issues

• Repurpose single price addition workflow to speed up single price / single picture case open-prices-frontend#961
• Multiple proof upload page open-prices-frontend#983
• Proof: add new "consumption-related" fields on receipts: total amount, items count #387 (comment)
• Leverage Hunger Games to crowdsource price addition (from price tag images) #523

[raphodn]

I would definitely be curious of a screenshot (or quick video ?) of your setup !

I presume you have 1 proof image per price in your case ? So it does require a bit of time to first upload the proof, and then add the single price. The current price addition workflow is indeed better for multiple prices.

This discussion would help in offloading the time spent per proof/price and leave it to the community. 3'40 per price is too long 😢

And we could use some image detection/cropping

• split images with multiple price tags, into images with only 1 single price tag, to make it more distributable on Hunger
• having OCR - for instance Proof: send to Google Cloud Vision, and store the matching JSON output #320 - would pre-fill the price & barcode forms, and have the users only "double check"

[raphodn]

fyi my current workflow :

• i take pictures with my phone (stored in the gallery), and add the proofs in the OP web app via the phone. my pictures always have at least 2 or 3 price tags on them
• once home, I open my laptop, connect to my dashboard, and for each proof add the related prices
  • my laptop screen is small, so sometimes by zooming on my phone image (phone next to my laptop)
  • sometimes opening the proof image in full size and displaying it side by side (but the web app compresses the images, the quality is not always enough...)

[raphodn]

@serpico

• I take pictures with my phone (stored in the gallery), for specific reasons I would rarely have 2 or 3 price tags in one picture and if I did I would only exploit the one I'm interested about, as you guessed it's almost always 1 price / 1 picture.
• Once home, I open my laptop ( 12-inch screen, so who's got the smallest one ? 🤣 ) , copy the pictures from the gallery to a specific folder, open the first picture with an image viewer ( it loads all the pictures in that folder, so I can easily navigate the pictures of the day )
• I also open a .txt file where I have most of my tracked products ( 1 line containing a short description and the EAN separated by several tabulation - ballpark 200 lines for products only interspersed with notes - somehow match my count of 183 displayed in my dashboard ) , ctrl+f of either the tail of the EAN or some character chain I know will give me the product I'm looking for so I can copy/paste EAN straight into the appropriate box when needed.

I'll try your method ( phone to upload pic + laptop to add the price(s) ) next week and time myself, see if it provides a sizeable improvement and report back.

PS: I have prepared a longer version ( also for myself ) detailing the specific reasons for 1 pic / 1 price, associated with a sample of a few dozens of pictures illustrating why a 1 pic / several prices, I think wouldn't work for me, I'm willing to send it to you on your email, if you want us to dig deeper.

[raphodn]

don't have much time to answer right now, but I'd be willing to dig deeper.

In a single shop & day session, how many pictures do you take ?

If there existed a multi-upload proof page (similar to https://prices.openfoodfacts.org/proofs/add/single - yeah needs to be fixed - but allowing to select multiple images for the same shop & date) would it help you ? though afterwards you would need to go to your dashboard > proofs > for each one add the corresponding price ^^ so only a marginal time gain.. ?

[raphodn]

@serpico

No worries, I took notes so we can postpone for later, whenever...

I'd say 15 to 40 pictures, not all will be uploaded ( doubled for quality )...but I'd say 10-12 prices per session minimum and as many pictures.

If there existed a multi-upload proof page (similar to https://prices.openfoodfacts.org/proofs/add/single - yeah needs to be fixed - but allowing to select multiple images for the same shop & date) would it help you ? though afterwards you would need to go to your dashboard > proofs > for each one add the corresponding price ^^ so only a marginal time gain.. ?

I think it'd help, but let's table that, first I need to try your workflow, see how it goes.

I'd really like to give you an embryo of a mock up but I need to really to decompose the process ( i.e : upload picture > copy the EAN from .txt > hit tab on OP > paste the EAN > ) I'll open another issue for me to work on that, if you could take a look at it whenever you can and tell me what you think.

Here is the issue in question 961

[serpico]

Here's some feedback after a few price logging sessions, using some of your advice regarding the workflow :

A/ First session

1/ Upload the proofs using the webapp on the phone ( same type, same folder for source image, same shop location, same date, same currency ) for all the 29 pictures, it took 20' so on average 41" per picture, including 13" really uploading ( when you hit the green button after selecting the file/image, the button is spinning and when it's over the "done" button is unlocked and you can go into the menu and start the next proof.

2/ Log The prices
I followed your workflow, once all the proofs are uploaded I go into the proof gallery, bottom menu of one proof card > Add a price

Hit "TYPE A BARCODE"
copy/paste the barcode
Hit enter
Click on the price input box ( to focus )
Type the price ( display the last known price for this product and this location and openfoodfacts/open-prices-frontend#567 could be useful )
Hit upload
Hit done

Here there's probably something to do in term of keyboard shortcuts, where the focus gonna be/should be every time we validate each step, there's probably a few keystroke, pointer movement to be saved...

19 prices in 18'42" or 1122" 59" per price

So combined, per proof+price, it averages 1'40" per price ( upload and logging ).

This is, what I consider, a significantly favorable sample as I trimmed my .txt file containing the EANs so I only have products from this shop and sorted in the order I go through the shop - there's a caveat here, the order in which I upload the proof might have an slight impact, it'd better to have the same order on the .txt file and on the proof gallery. I also keep the new products ( the ones that are not on my EAN trimmed list/.txt file ) as process separately - and it takes a lot more time ( change the quantity on OFF, add the brand, create the product from scratch...), but it's significant in my case, I'd say for every 15 to 20 known product ( allow fast mode, copy paste EAN and that's it ) there are 3 to 5 which require the slow way, I have a ballpark somewhere, I'll try to look for it

B/ Second session

1/ I copied the proof pictures from the phone to the laptop in a dedicated folder, then upload the proofs using the browser of the laptop instead of the doing on the phone like in A/.
There's a noticeable improvement to upload x16 proofs it took 8'44" or 524" or 33"per upload ( this is including the step to create the folder in the laptop, copy from phone to laptop )

2/ log the prices : x13 prices in 12'17" soit 737" ou 57" per price

Combined 1'30" per proof+price ( upload and logging )

C/ Third session

1/ Same method as in B/ ( copy pictures from phone to laptop, then upload from laptop )
upload 17 proofs ( some duplicates ) in 8'20" or 500" averages to 29" per proof

2/ log the prices : 11'40" for 11 prices or 700" averages 64" per price

Combined 1'33" per proof+price ( upload and logging )

One thing that still slows the process, is once the price is sent the dashboard page is automatically loaded, then you need to click the Proof button at the end of each price, load the gallery, look for the next proof - if we went back directly to the proof gallery we could save some time - even more if there was an option to display only the proofs with 0 price (having a "stack/deck" of proofs to go through for one session, the stack would slim down every time we log price - only proofs with 0 price would stay in the "stack/deck" to be exploited - so if you mess up the upload order compare to your list of EANs and proceed in reverse order, each time you need to load 1 or more block of X proofs - a more desirable situation would be the first ( at the top ) proof you load/see in the proof gallery is the one associated with the next EAN on your list.

Hope it's not too messy.

[TTalex]

Related: I've noticed losing a bit of time when mistyping/misreading barcodes, so I've proposed a solution here: openfoodfacts/open-prices-frontend#991

Note that this doesn't really help on the above feedback. It might even worsen it, since it breaks some of the enter key shortcuts.

[raphodn]

Can you explain a bit more ? You mistype/misread the barcode because the image is not very clear, or your hesitating between a 3 & a 6 or 8 for instance ? (edit: ok read your PR description, there's more details there 👍 )

I opened an issue in search-a-licious a few weeks ago to see what could be done for the fuzzy search, probably related :)
openfoodfacts/search-a-licious#242
--> these complex product barcode queries could be sent to search-a-licious instead of OP backend (but we wouldn't have the price_count)

[raphodn]

With the recent "Contribution assistant" built by @TTalex , I'm seeing 2 (or 3) seperate steps that we could isolate, allowing distribution on multiple users.

Some of these thoughts have already been shared in a similar discussion - #523 - and some related issues.

Step 1: upload 1 or multiple proofs [user]

Speed up proof addition when a user has multiple proofs for a given location & date.

Related issues

• Multiple proof upload page open-prices-frontend#983

Step 2: extract data from images [AI/ML]

• automatically predict the proof type
• automatically crop out price tags (bounding boxes)
• ask an AI to return the price predictions (see the current Contribution assistant that uses Gemini AI)

Related issues

• Proof : auto classification on user upload #394
• Proof: send to Google Cloud Vision, and store the matching JSON output #320
• AI-powered price addition workflow (Contribution assistant experiment) open-prices-frontend#1044
• Create a proof_prediction table to store predictions from ML models #511
• Add a price_tags table to store predictions #611

Step 3: ask users to validate the price predictions [user]

Last step, we have price info that has been extracted from the images, we need a user to validate and make sure that it is the correct product (or raw category) and price.

Related issues

• Proofs (Price tags): allow users other than the owner to add prices #568

[raphodn]

Quick update :

1. in the frontend the multiple proof upload page is live since a few days

• https://prices.openfoodfacts.org/proofs/add/multiple
• it's missing a check on the images (and some info message) to make sure they are all from the same date

2. in the backend we've unlocked 2 things

• proofs can now be uploaded anonymously
• any user can add prices on PRICE_TAG proofs

Next step:

• frontend: a new contribution page displaying all PRICE_TAG proofs without any prices. And allowing any user to start adding prices (decided to move faster and go directly to the price-level instead of proofs)
• frontend: start splitting your price dashboard, between personal data (RECEIPT/GDPR proofs & their related prices), and common data (PRICE_TAG proofs or prices)
• backend: store price-level predictions
• frontend: a new contribution page displaying price tags to be validated (and effectively create prices from predictions)

[raphael0202]

Agree! I would also add as next step, add a multiple image upload form accessible without log-in for anonymous proof upload (as discussed last wednesday).

[raphodn]

fyi i updated the workflow, to show the intermediate "price tag" prediction

[raphodn]

Most of the work has been done in the backend during the last few days !!
#611

[raphodn]

I just created an issue in the frontend to start working on a "price tag validation" UI
(restricted to only your proofs/prices for now)
openfoodfacts/open-prices-frontend#1137

[serpico]

Should we split the discussion ? as :

• Speed up proof & price addition using AI
• Speed up proof & price addition by refining the UI [ full manual - one of the scenario I'm thinking about is the picture is so bad the person that took it can barely make out the EAN ( still can but also by trying to amend/correct the EAN and cross reference/check using brand, shop, whatever ) so what's AI gonna do...]

Example : In the above workflow I'd, personally, would put "Classify receipt/price tag" on the 1st step since when I have a batch of price/pictures to upload I do them all, without receipt in between...

[raphael0202]

Even if the EAN cannot be read, we can still provide product suggestion based on :

• the price tag label (ex : confiture mûre 400g Grosjean)
• the product that is associated with the price tag: we can find similar products in OFF based on the product image

We can't guarantee a 100% match rate, but at least we could provide valuable suggestions most of the time.
When ML fails, we should always let the user provide the data efficiently. ML-based contribution and manual contribution should be thought as a whole in my opinion.