Move to Golang?

[jef]

Description

There are many reasons why we would either stay with TypeScript or move to Golang. I want to describe a few pros and cons and leave this up to discussion.

Pros

• Publishing packages
  • No more downloading Node.js/Git for those who aren't familiar
  • Still have the option to download golang/git for contribution and nightlies
• More lightweight
• Probably faster
• Pretty awesome scraper (https://github.com/gocolly/colly)

Cons

• Potentially less contributors
• Possible build problems (doesn't work on my machine)
  • We kind of get a lot of these already, so not sure if a drawback
• Regressions
• A lot of work without a ton of benefit

---

I'd like to get some feedback and if enough contributors are fans of this, perhaps we'll start refactoring.

I already have some refactoring in mind for stores, but I'd like to hold off on that if we decide to move to Golang.

[njgreb]

Is it worth setting up a discord or slack for this project for discussions and sharing or do we want to keep it all inside github?

[jef]

Is it worth setting up a discord or slack for this project for discussions and sharing or do we want to keep it all inside github?

I have one in the README if you'd like to have faster communication in there. I think this will be good to serve as somewhat of an audit as well for those who don't join.

[thetimothyp]

I haven't used Go so I don't have any input as far as pros/cons, but I'll say I'd be interested in working in a Go codebase from a learning perspective.

[ysmood]

If you want to replace puppeteer, this project could be a more powerful choice: https://github.com/go-rod/rod

[v1ct0rv]

Description

There are many reasons why we would either stay with TypeScript or move to Golang. I want to describe a few pros and cons and leave this up to discussion.

Pros

• Publishing packages

  • No more downloading Node.js/Git for those who aren't familiar
  • Still have the option to download golang/git for contribution and nightlies

• More lightweight

• Probably faster

• Pretty awesome scraper (https://github.com/gocolly/colly)

Cons

• Potentially less contributors

• Possible build problems (doesn't work on my machine)

  • We kind of get a lot of these already, so not sure if a drawback

• Regressions

• A lot of work without a ton of benefit

I'd like to get some feedback and if enough contributors are fans of this, perhaps we'll start refactoring.

I already have some refactoring in mind for stores, but I'd like to hold off on that if we decide to move to Golang.

I like the idea in order to gain performance and less resource consumption.

About the cons regarding to build problems, we should release binaries for each platform, so will be easy for the end user to run the app.

To avoid possible problems in regression we should write unit/integration tests.

I don't have a lot of experience in go but this will be a good opportunity to learn go in depth.

[jef]

About the cons regarding to build problems, we should release binaries for each platform, so will be easy for the end user to run the app.

Done relatively easy with Go build toolkit. Never got it to work with Node.js using pkg

[benw25]

I'd prefer sticking with node/JS. The cons you have listed don't seem to outweigh the pros. The pros themselves aren't even that substantial:

• Publishing packages: Downloading nodejs / Git / Docker is a slight headache for non-devs, and it would be nice to use a package instead; however, I believe that there should be some necessary setup even outside of these packages, namely a VM / server, ideally with a VPN & killswitch so that the user doesn't get blocked or flagged. Setting those up is non-trivial for non-devs (and is highly suggested from my point of view), and there is no easy way to do that automatically. So asking to install node/git/docker on top of a VM is not a huge ask or a huge drawback in my opinion, especially since most modern webapps require those (and sometimes more)
• Performance & size improvements are nice, but not worth the work to rewrite everything
• The Colly scraper looks great

Cons:

• A lot of work without a ton of benefit
• Having to maintain products in two different languages, potentially with different features, issues, and bugs in each. You may get more reports & bugs using with the new app because of the ease of installation, and combining that with potentially less contributors sounds like a headache to maintain

[benw25]

That being said, ultimately I believe that the biggest factor should be where you want to take this project. This a much larger factor than any of the pros/cons that 3rd parties have listed.

If you want the widest userbase, Golang.
If you want the least maintenance & want more time to explore other projects or start new projects, Node/JS.
If you want to explore that cool scraper and see where that takes you, Golang.
If you want to refactor and instead just prefer to rewrite the whole thing, Golang.
If you want to expand on current features in the quickest amount of time possible (and don't think it requires a rewrite/refactor), Node/JS.

[jef]

A lot of work without a ton of benefit

If you want to expand on current features in the quickest amount of time possible

If you want the least maintenance & want more time to explore other projects or start new projects

All very valid and fall under this category to me.

So asking to install node/git/docker on top of a VM is not a huge ask or a huge drawback in my opinion

That makes me feel better 😄

---

Very thoughtful feedback, @benw25. I appreciate your input.

[MichaelGatesDev]

I'd like to point out that there are libraries like pkg which allow you to distribute a single executable which can still use outside resource (e.g. config file). Your users wouldn't need to download Node or any dev tools as they get bundled as needed. The downside is that the executable size is ~100MB.

[jef]

Yeah, I played with pkg at some point. It actually ended up being around 500MB 😮 Conceptually it's a great idea, just wish it was smaller. I blame Chrome being bundled into it. I think we could probably set it up to use an installed version of Chrome instead of downloading Chromium.

[dmsolutionz]

@jef Great idea, the only downside I see is that typescript / node is widely used with many more opportunities to have more people contribute. Alternative idea; could move the app to electron and just make it simply configurable for people, should reduce the noise on "why its not working on my machine". Thoughts?

[sporaktu]

You know, to go along with the electron idea, a simple SQLite database linked to a form with all the possible options could eliminate the need for a .env file. Or, if you wanted to go even lighter (though likely not faster), a simple fs read/update of a JSON file with the user's settings could do the same thing. Just set that options page as the first thing a user sees and then put a little settings icon somewhere to open it again.

I'm not sure how far you want to go with this, but the JSON route would also make it pretty trivial to update the options with a simple API query if functionality is added or changed. It would be pretty cheap to host just the settings endpoint somewhere too, especially if it was getting pinged once per user/per day, it likely wouldn't go over a free tier of many hosting sites like Heroku, and it could probably run on a free tier AWS EC2 server.

[jef]

the only downside I see is that typescript / node is widely used with many more opportunities to have more people contribute.

Great point. By now, I think we will stick with TypeScript. I also believe this is the main reason why this project has grown to this size.

Alternative idea; could move the app to electron and just make it simply configurable for people, should reduce the noise on "why its not working on my machine". Thoughts?

I don't have any experience with this, but would be awesome for packaging.

---

Or, if you wanted to go even lighter (though likely not faster), a simple fs read/update of a JSON file with the user's settings could do the same thing

I'm actually working on this right now! It's all YAML based and requires no config at a minimum. Currently it should work with all the configuration outside of store related stuff.

Eventually, I'd like to create a server that would add contributed links through a simplified view and go through approval that way. Basically meaning that streetmerchant can either use local config files or pull from an API.

So you're basically right on par with where I think the next best move is! Look out for the branch prerelease/4.0.0 -- that's where you'll find those changes.

[Black-Widow]

So you're basically right on par with where I think the next best move is! Look out for the branch prerelease/4.0.0 -- that's where you'll find those changes.

@jef I can't find the branch, is it normal?

[jef]

I haven't published it yet. I will post again when I do!

[Black-Widow]

I haven't published it yet. I will post again when I do!

oh ok sorry. thx

[jef]

Published. Know that it is instable and has no documentation on how to use yet. I'll continue to work on it.

Thanks!

---

From here, let's keep this issue related to golang. Otherwise, feel free to browse https://github.com/jef/streetmerchant/issues/1217 and add issues to it.