Meteor 3.0: Adding Async Methods to server #884

bratelefant · 2023-12-21T20:18:35Z

This is a first try on async methods as a PR draft. Updated eslint and mocha testing as well; core.js and cursor.js were a starting point. Work is progressing on server.js.

Related to #865

@Prinzhorn

📦 v2.2.0 __Breaking Changes__ - ⚠️ Changes in `namingFunction`, — now naming function acts the same on the Client and Server, upon insert, load, and write. Test your implementation with changed logic. Output of Server function supersedes Client's function output __Changes__ - 📔 Merge veliovgroup#843 and fix veliovgroup#820, thanks to @Prinzhorn - 📔 Documentation refactoring focused on examples and its simplifications - 👨‍💻 Support nested custom path returned from `namingFunction` - 👨‍💻 Fix `namingFunction` behavior on Client and Server in upload, load, and write methods, closing veliovgroup#842; Thanks to @chrschae - 👷‍♂️ Now library exports its helpers `import { FilesCollection, helpers };` - 👷‍♂️ Add `.meteorignore` to minimize package's footprint

@chrschae

📦 v2.2.1 - 👨‍💻 Fix veliovgroup#842, a newly detected bug by @chrschae; Fixing case when `namingFunction` returns new nested path cause exception in `.write()` and `.load()` methods

@xet7

📦 v2.3.0 __New features:__ - ✨ `opts.sanitize` method, read more in [*Constructor* docs](https://github.com/veliovgroup/Meteor-Files/blob/master/docs/constructor.md); Thanks to @xet7 and @mfilser __Other Changes:__ - 👷‍♂️ Minor codebase enhancements and cleanups

@Leekao

v2.3.1 __Changes:__ - 👨‍💻 Improve `createIndex` helper - 👨‍💻 Improve error output when FileSystem destination not writable; Related to veliovgroup#857, thanks to @Leekao - 🐞 Fix custom `allowedOrigins` option for CORS; Closing veliovgroup#850, thanks to @djlogan2; __Notes:__ - 👨‍🔬 Tested with latest release of `[email protected]`

📔 Improve AWS S3 documentation

📦 v2.3.2 - 👨‍💻 Potential fix for veliovgroup#857 (windows)

👷‍♂️ Update .eslintrc

@Gobliins

📦 v2.3.3 __Major changes:__ - no __Changes:__ - 👨‍🔧 Fixed veliovgroup#870, thanks to @Gobliins - 🤝 Compatibility with `[email protected]`

bratelefant · 2023-12-27T22:26:18Z

Hi guys, please comment on this first attempt. In order to better understand what's going on I decided to add some tests for the server part, FilesCollection, core and cursor. Unfortunately this adds overall dependencies, especially in package.js; if anyone knows how to limit these dependencies to dev / testing, pleas let me know. Also tweaked the .eslintrc a little bit. Core and cursor now additionally have async methods. Work on server.js is in progress.

I suggest that the async versions of all the methods do not use any callbacks but throw errors instead and return promises resolving the results, cf. FilesCollection.addFileAsync for example.

Which callbacks need async versions? E.g. for my project, I need an async version of protected(), since I do policy checks against the db here, which will be async as well of course. I guess an async version onAfterUpload could be helpful, since the aws sdk v3 for example also uses async functions to send putObject requests etc.

dallman2 · 2024-03-26T05:39:41Z

Will try this update tomorrow and let you know how it goes (I'm on Pacific coast time :P). I've been deep into css modules all day so I'm going to have to shift a bit to deal with the files collection.

dallman2 · 2024-03-28T22:31:38Z

No issues for me using beta.5. That is to say, my app starts and runs. Have not done totally comprehensive testing.

harryadel · 2024-04-29T07:14:37Z

@bratelefant @dr-dimitru Maybe it's time to release a new beta that relies on the latest RC instead of beta?

jankapunkt · 2024-04-29T09:15:46Z

A beta release would be awesome so I can do some integration tests with our projects and give you early feedback

dr-dimitru · 2024-04-29T10:19:42Z

@harryadel @bratelefant @jankapunkt 3.0.0-beta.6 is out with support for [email protected]

harryadel · 2024-04-29T10:36:07Z

Thanks for acting so quickly @dr-dimitru 👏

harryadel · 2024-05-21T14:55:10Z

Can you make onBeforeUpload on the client side async please?
https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

cc @bratelefant @dr-dimitru

make-github-pseudonymous-again · 2024-05-23T14:13:33Z

Can you make onBeforeUpload on the client side async please? https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

cc @bratelefant @dr-dimitru

I think in general all hooks should be async. Any of them might require interacting with the database which is now something that is async by default.

make-github-pseudonymous-again · 2024-05-23T16:14:36Z

@bratelefant My only pain point so far with the current state of the PR is the lack of documentation: took me some time to figure out that the callback version of Meteor.write was gone. The error you get by trying to use the old signature on the new one is just a timeout, if you are lucky to have configured one (tests). This PR could have been done without this change, right? If we are aiming at a new version of ostrio:files without any callback-style functions, then we should also change the client signatures (insert, etc.).

bratelefant · 2024-05-23T16:32:37Z

Can you make onBeforeUpload on the client side async please? https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

That'd be definitely one major point to add to this pr. Did all changes with focus on the server side, client only methods did not get touched.

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

Yes, in our discussion here at some point we decided to make this an update with breaking changes by removing all sync methods on the server side.

@make-github-pseudonymous-again I tried to keep documention in the server side method comments aligned with the code changes, but that was definitely not checked agains tests / code.

I would be really grateful if someone could jump on the bandwagon and contribute to the PR, I am having a pretty 'crunchy' time at work atm

make-github-pseudonymous-again · 2024-05-24T07:01:06Z

server.js

+    let _id;
+    try {
+      _id = await this.collection.insertAsync(helpers.clone(result));
+      try {
+        await this._preCollection.updateAsync({_id: opts.fileId}, {$set: {isFinished: true}});
+        if (_id) result._id = _id;


Is it conceivable that this id-setting logic sometimes fail? I have an existing test that yields fileRef._id = undefined in the on('end') handler of FileCollection#insert (client-side call)`.

onUploaded is even worse: the fileRef is simply undefined.

@bratelefant do you think we should address this as part of this PR?

I'm getting similar undefined results after an upload via

ContactFileCollection.insert({ file: file, }).on('uploaded', (uploadError, file) => { console.log('===== uploaded', uploadError, file)

I would adress it in a separete PR if it will not interfere with this one (merge conflicts etc.) otherwise directly here

…oreUpload

StorytellerCZ · 2024-09-25T15:22:37Z

Any update on this?

harryadel · 2024-09-26T09:35:23Z

Hello, I created a PR to allow us to use async onBeforeUpload bratelefant#2

Make onBeforeUpload async compliant

hluz · 2024-10-08T21:45:09Z

Hi everyone... Not trying to be pushy, but this is one of the few remaining hurdles in migrating to Meteor 3 for many of us. Even if it was only a new beta version adding the async onBeforeUpload changes would help... @dr-dimitru @StorytellerCZ @jankapunkt

wolasss · 2024-11-14T10:56:06Z

Hey, any updates on this ?

jankapunkt · 2025-01-04T08:59:26Z

Hey @wolasss @hluz @bratelefant I can take a look at this finally... may need some time though to test everything with our existing staging projects

jankapunkt

First of all big thanks for this PR and the all the work on this. I am currently creating a repro to test all this with a GridFS setup but it looks good for minimal functionality.

However, I found a few thinks which I commented each in the respective lines.

@dr-dimitru I think there is still the big question if there is the intent to support Meteor 2 and 3 or doing a clean break and just go for Meteor 3 and onwards.

Besides that I found there are much more things to be done after merging this pR:

docs need to be updated
server.js should be refactored into smaller files to reduce mental load during debugging sessions

@dr-dimitru if you have time we could get in contact this week and shortly talk about how we can move forward with this one. I'm willing to finish the great work that has been done here but I first think we need a clear mission for a stable 3.x release

jankapunkt · 2025-01-04T15:49:32Z

client.js

+    const uploadInstance = new UploadInstance(config, this);
+    if (autoStart) {
+      uploadInstance.start().catch((error) => {
+        console.error('[FilesCollection] [insert] Error starting upload:', error);


Will this error also go into onError on the client in order to properly inform users with human readable error?

jankapunkt · 2025-01-04T15:49:54Z

.github/workflows/lint.yml

+    - name: setup node
+      uses: actions/setup-node@v3
+      with:
+        node-version: '14.x'


Should bump to latest Node LTS (22)

jankapunkt · 2025-01-04T15:50:10Z

.github/workflows/lint.yml

+    runs-on: ubuntu-latest
+    steps:
+    - name: checkout
+      uses: actions/checkout@v3


All GitHub core actions are in the meantime at version 4

jankapunkt · 2025-01-04T15:51:11Z

.github/workflows/testsuite.yml

+      - name: Setup meteor
+        uses: meteorengineer/setup-meteor@v1
+        with:
+          meteor-release: '2.14'


Does it make sense to add Matrix tests here? In Meteor Community Packages we do Matrix tests for Major Meteor versions 2.x and 3.x if packages have to support both

jankapunkt · 2025-01-04T15:56:18Z

cursor.js

-    return this.cursor.count();
+  async countAsync() {
+    this._collection._debug('[FilesCollection] [FilesCursor] [countAsync()]');
+    return this.cursor.countAsync();


while this is good backwards compatibility the prefereed method for counting docs is now async collection.countDocuments and async collection.estimateDocumentCount which are missing in the core.js implementation of FilesCollection

jankapunkt · 2025-01-04T16:38:51Z

server.js

-          fs.writeFileSync(opts.path, '');
-        }
+    // check if the file already exists, otherwise create it
+    let mustCreateFileFirst = false;


this seems to be redundant with the code in line 1278++ and could be extracted into a method

jankapunkt · 2025-01-04T16:40:28Z

server.js

-              }
+      if (opts.timeout > 0) {
+        timer = Meteor.setTimeout(() => {
+          controller.abort();


Note, that Node 14 has no native AbortController implementation and a polyfill is needed; so this might cause issues when we want to have backwards compat with 2.x

jankapunkt · 2025-01-04T16:41:56Z

server.js


-    return this;
+      if (!result.size) {


if this is newly added behavior then please add a few comments. also - is pipeline available in node 14?

jankapunkt · 2025-01-04T16:43:46Z

server.js

+      const fileRef = await this.collection.findOneAsync(_id);
+
+      if (proceedAfterUpload === true) {
+        this.onAfterUpload && this.onAfterUpload.call(this, fileRef);


missing await

jankapunkt · 2025-01-04T16:46:24Z

server.js

-    if (files.count() > 0) {
-      files.forEach((file) => {
+    if (await files.countAsync() > 0) {
+      await files.forEachAsync((file) => {
        this.unlink(file);


this is debatable but if you want to use sequential for each async then you have to return the promise, generated by this.unlink in order to have the underlying await to take effect.

otherwise I don't know if this has any implications, like race condition etc.

dr-dimitru · 2025-01-05T16:55:30Z

@jankapunkt thank you for your review, I'll go though it, and see how much I can fix, then we decide if call is needed, looking forward to new beta release

jankapunkt · 2025-01-16T07:39:57Z

@dr-dimitru if you want you can merge this into migration/3.0 branch and I can continue to work on the things I commented

dr-dimitru and others added 25 commits June 29, 2022 16:08

Merge pull request veliovgroup#845 from veliovgroup/dev

313e842

📦 v2.2.1 - 👨‍💻 Fix veliovgroup#842, a newly detected bug by @chrschae; Fixing case when `namingFunction` returns new nested path cause exception in `.write()` and `.load()` methods

Merge pull request veliovgroup#859 from veliovgroup/dev

df7a6c9

📔 Improve AWS S3 documentation

Merge pull request veliovgroup#860 from veliovgroup/dev

4f58852

📦 v2.3.2 - 👨‍💻 Potential fix for veliovgroup#857 (windows)

Merge pull request veliovgroup#864 from veliovgroup/dev

b312ef9

👷‍♂️ Update .eslintrc

Merge pull request veliovgroup#871 from veliovgroup/dev

49f3bed

📦 v2.3.3 __Major changes:__ - no __Changes:__ - 👨‍🔧 Fixed veliovgroup#870, thanks to @Gobliins - 🤝 Compatibility with `[email protected]`

implement async methods

93b7a73

getLink async

a6c6da3

lint fix

926e8e2

fix Match.OneOf etc lint rule

c811a75

tiny tests npm script

b37aba8

added some mocha testing

f7bbba1

script for mocha

4af4af6

Merge remote-tracking branch 'original-meteor-files/dev'

aac6555

tests

e502124

few more tests

91fbf48

link() is not async

c7fe2a3

more tests, remove tests fails

5c31934

fixed tests

743c12f

rest of cursor tests

ff30780

few more tests

1ea4e38

Add devDependencies and update removeAsync method

bf63873

addFileAsync

e727ae8

bratelefant added 4 commits December 28, 2023 01:57

test for download

3edc652

downloadAsync tests

f97b4a9

prepareUploadAsync

497e8c2

_finishUploadAsync and tests

2dfa4de

harryadel mentioned this pull request Apr 1, 2024

[WIP] Meteor v3 Meteor-Community-Packages/meteor-collection-hooks#309

Merged

bratelefant mentioned this pull request Apr 18, 2024

User can not be determined in protected callback in Firefox #888

Open

make-github-pseudonymous-again reviewed May 24, 2024

View reviewed changes

dr-dimitru changed the title ~~Adding Async Methods to server~~ Meteor 3.0: Adding Async Methods to server Jun 28, 2024

make-github-pseudonymous-again mentioned this pull request Jul 5, 2024

Upgrade Meteor to 3.0 infoderm/patients#995

Draft

donstephan mentioned this pull request Jul 29, 2024

[3.0.0-beta.6] fileObj._id on upload end ? #889

Open

bratelefant and others added 3 commits August 7, 2024 11:47

return fileObj in uploaded and end event

d573741

Initial commit

22969a0

Merge branch 'master' of github.com:harryadel/Meteor-Files into onBef…

ed9e253

…oreUpload

Merge pull request #2 from harryadel/onBeforeUpload

0d6707e

Make onBeforeUpload async compliant

hluz mentioned this pull request Nov 5, 2024

Version 3.0.0-beta.6 onBeforeUpload does not support async db calls #892

Open

xet7 mentioned this pull request Nov 27, 2024

Wekan Windows 64bit version On-Premise wekan/wekan#5591

Open

jankapunkt requested changes Jan 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meteor 3.0: Adding Async Methods to server #884

Meteor 3.0: Adding Async Methods to server #884

bratelefant commented Dec 21, 2023

bratelefant commented Dec 27, 2023

dallman2 commented Mar 26, 2024

dallman2 commented Mar 28, 2024

harryadel commented Apr 29, 2024

jankapunkt commented Apr 29, 2024

dr-dimitru commented Apr 29, 2024

harryadel commented Apr 29, 2024

harryadel commented May 21, 2024 •

edited

Loading

make-github-pseudonymous-again commented May 23, 2024

make-github-pseudonymous-again commented May 23, 2024

bratelefant commented May 23, 2024

make-github-pseudonymous-again May 24, 2024 •

edited

Loading

make-github-pseudonymous-again May 24, 2024 •

edited

Loading

dr-dimitru Jun 28, 2024

67726e Aug 27, 2024

jankapunkt Jan 4, 2025

StorytellerCZ commented Sep 25, 2024

harryadel commented Sep 26, 2024

hluz commented Oct 8, 2024

wolasss commented Nov 14, 2024

jankapunkt commented Jan 4, 2025 •

edited

Loading

jankapunkt left a comment

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

jankapunkt Jan 4, 2025

dr-dimitru commented Jan 5, 2025

jankapunkt commented Jan 16, 2025

Meteor 3.0: Adding Async Methods to server #884

Are you sure you want to change the base?

Meteor 3.0: Adding Async Methods to server #884

Conversation

bratelefant commented Dec 21, 2023

bratelefant commented Dec 27, 2023

dallman2 commented Mar 26, 2024

dallman2 commented Mar 28, 2024

harryadel commented Apr 29, 2024

jankapunkt commented Apr 29, 2024

dr-dimitru commented Apr 29, 2024

harryadel commented Apr 29, 2024

harryadel commented May 21, 2024 • edited Loading

make-github-pseudonymous-again commented May 23, 2024

make-github-pseudonymous-again commented May 23, 2024

bratelefant commented May 23, 2024

make-github-pseudonymous-again May 24, 2024 • edited Loading

Choose a reason for hiding this comment

make-github-pseudonymous-again May 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StorytellerCZ commented Sep 25, 2024

harryadel commented Sep 26, 2024

hluz commented Oct 8, 2024

wolasss commented Nov 14, 2024

jankapunkt commented Jan 4, 2025 • edited Loading

jankapunkt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dr-dimitru commented Jan 5, 2025

jankapunkt commented Jan 16, 2025

harryadel commented May 21, 2024 •

edited

Loading

make-github-pseudonymous-again May 24, 2024 •

edited

Loading

make-github-pseudonymous-again May 24, 2024 •

edited

Loading

jankapunkt commented Jan 4, 2025 •

edited

Loading