[RFC] prepare migration of meta-qcom (and merge with meta-qcom-hwe)

[ndechesne]

Related to: Linaro/meta-qcom#680

This is probably impossible to properly review in Github. The key commits to review are the most recent commits in the PR branch.

First I am doing a 'non trivial' merge of meta-qcom (master) into meta-qcom-hwe (main) using "git merge --allow-unrelated-histories". Most of the merge is trivial with no conflicts, e.g. merging recipes, conf files, ... from both layers, just fine. The main conflicts were with README.md and conf/layer.conf. For both of these files, I have done a manual merge/edit so that they look right to me. These files need to be reviewed carefully.

The next 2 commits are changes in the Github workflows. What I am trying to do with the workflows is:

• For all stable up to styhead, they predate the migration, so Linaro will continue to maintain them, and the CI should be based on existing Linaro CI
• For master and all future stable branches, we are moving to Qualcomm CI

Note: the Linaro CI is not expected to work for now, since we will need to setup the secrets/token.

The result of the merge represents a very decent history of meta-qcom with ~1650 commits from ~50 developers over the last 10 years.

In terms of process, we eventually want to migrate meta-qcom into github.com/quic-yocto org. We can decide to merge this PR (when it's tested/ready) and continue the development of meta-qcom-hwe (main branch) here after the initial merge. When the repo is migrated, we will just need to do a final merge of the meta-qcom-hwe main branch into meta-qcom repo.

[lumag]

@ndechesne are there any remaining blockers? It would be nice to sort it out and complete migration.

[ndechesne]

@lumag I am traveling this week, and I don't have a lot of free cycles to finish. I am hoping we can make more progress next week..

I found some issues with the CI patches I made. I think we need more changes there. Since the 'master' branch is going to run on the QCOM CI, we need to move all Linaro CI outside of the master branch. So all branches up to styhead will be 'linaro' stuff, and master and future releases branches will be on QCOM CI. I think we should move all Linaro workflow either to styhead branch or perhaps scarthgap branch since it's the 'last' LTS branch on Linaro CI.

e.g. have all the workflow yaml files in scarthgap, and update all other branches to point to scarthgap workflows. Currently scarthgap and styhead have a copy of the workflows, kirkstone refers to workflow from 'master' branch (with: uses: Linaro/meta-qcom/.github/workflows/build-template.yml@master). and older branches (honister and before) are broken, as they refer to workflow in ndechesne/meta-qcom ( uses: ndechesne/meta-qcom/.github/workflows/build-template.yml@master)..

So do you want to get the Linaro CI updated/fixed in all branches other than master first?

I am planning to the Linaro CI workflow from this PR (when I get back to it)

[lumag]

I completely forgot about that. I will take a look at the CI for other branches

[koenkooi]

FWIW, a kas based build for qcs6490-rb3gen2-core-kit using this PR builds without issues.

[lumag]

@ndechesne I think I've fixed all outstanding CI issues.

[igoropaniuk]

Taking into consideration that there are currently 113 commits in meta-qcom-hwe and it's not currently "used in production" (in the terms of maturity), is there any reasons why the decision was made to merge it this way and not the other way around: fork meta-qcom first and then merge meta-qcom-hwe into it? This would definitely have required less effort in merging and follow-up testing, and as a result we would end up with 1:1 replication of meta-qcom git history

Another concern is that as for meta-qcom-hwe "rebase and merge" strategy was chosen from the very beginning, and pulling all merge commits from meta-qcom IMO might "spoil" existing git history in meta-qcom-hwe (I know it's just a matter of preference, but anyway there will be some chaos). Have we considered getting rid of merge commits (I know that it's a huge amount work, but just curious if it's feasible)?

[ndechesne]

Taking into consideration that there are currently 113 commits in meta-qcom-hwe and it's not currently used in production, is there any reasons why the decision was made to merge it this way and not the other way around: fork meta-qcom first and then merge meta-qcom-hwe into it? This would definitely have required less effort in merging and follow-up testing, and as a result we would end up with 1:1 replication of meta-qcom git history

git merge A from B, or git merge B from A. I don't see how it makes the merge any different. The same merge conflicts would exist. And in fact, there really was no difficult merge at all. It was fairly trivial to do the merge, the way I did it.

Since we are merging the 2 branches, we still keep the history of both branches, so the history of meta-qcom is replicated. Can you please elaborate? I am open to doing things in a different way if that makes anything simpler.

Another concern is that as for meta-qcom-hwe "rebase and merge" strategy was chosen from the very beginning, and pulling all merge commits from meta-qcom IMO might "spoil" existing git history in meta-qcom-hwe (I know it's just a matter of preference, but anyway there will be some chaos). Have we considered getting rid of merge commits (I know that it's a huge amount work, but just curious if it's feasible)?

That is correct. the history is going to be a bit of mess, but moving forward we will adopt rebase and merge again. As long as we kept the 2 branches history in the merge, I think it's fine.

[igoropaniuk]

@ndechesne

Since we are merging the 2 branches, we still keep the history of both branches, so the history of meta-qcom is replicated. Can you please elaborate? I am open to doing things in a different way if that makes anything simpler.

I got confused as I was thinking about rebase for some reason when I wrote "1:1 replication of meta-qcom git history", however you explicitly mentioned "merge" in the PR description, so sorry for the nohistoryise.
My point was that having PR with 112 commits would look more lightweight and less scary :) , but I do agree that in the end we would get the same git tree and it's just a matter of preference

[koenkooi]

When using 'kas build' for the rbd3, this doesn't seem to generate a qcomflash tarball anymore.

[igoropaniuk]

@koenkooi I think that's because it doesn't include this commit:

commit d6358019cce14f21e95a6a31cec28b134e2fa0d7
Author: Ricardo Salveti <[email protected]>
Date:   Thu Nov 14 19:22:20 2024 -0300

    image_types_qcom: add support for qcomflash IMAGE_FSTYPE
    
    Add support for a new IMAGE_FSTYPE which creates a single tarball
    containing all the relevant files to perform a full clean flash,
    including partition metadata, boot firmware, ESP partition and the
    rootfs.
    
    Signed-off-by: Ricardo Salveti <[email protected]>

[ndechesne]

I updated the PR. What's changed:

1. I did the merge again with the most recent commits from meta-qcom and meta-qcom-hwe
2. completely disabled (removed the workflow) the Linaro CI in main
3. added meta-qcom generic machines in CI

[lumag]

Nice! Ack from my side.

[ndechesne]

@ricardosalveti when we decide to merge this PR, we will need to do a merge commit, it's an exception to our process, but i believe that a rebase/squash will break everything..