arrow

ARROW-12846: [Release] Reduce download/upload bandwidth for APT/Yum repositories

Summary:

Generating metadata for APT/Yum repositories from all .deb/.rpm
on local from scratch

    ||
    \/

Generating metadata for APT/Yum repositories only for built .deb/.rpm
and merging the generated metadata and existing metadata
(Released .deb/.rpm aren't downloaded.)

Before:

  For RC:
    1. Download built .deb/.rpm for RC
    2. Sign the built .deb/.rpm
    3. Upload the signed built .deb/.rpm to
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc
    3. Download all released .deb/.rpm from
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}
    4. Download the all built and signed .deb/.rpm from
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc
    5. Generate metadata for APT/Yum repositories with local .deb/.rpm
       from scratch
    6. Upload the metadata for APT/Yum repositories to
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc

  For release:
    1. Download the all .deb/.rpm and metadata for APT/Yum repositories from
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc
    2. Upload the downloaded files to
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}

After:

  For RC:
    1. Download the built .deb/.rpm for RC
    2. Sign the built .deb/.rpm
    3. Generate metadata for APT/Yum repositories for the built and
       signed .deb/.rpm
    4. Download metadata for APT/Yum repositories from
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}
    5. Merge metadata of 3. and 4.
    6. Upload the built and signed .deb/.rpm and the merged metadata
       for APT/Yum repositories to
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc

  For release:
    1. Download only uploaded files by 6. for RC from
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}-rc
    2. Upload the downloaded files to
       https://apache.jfrog.io/ui/native/arrow/{debian,centos,...}

Closes #11263 from kou/release-linux-partial-upload

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>