While trying to optimise a composer flow I run into this issue.
Every time I run composer update -vvv I can see that there is a set of 10 modules that always gets rewritten into cache.
Composer version is 1.8.6
[15.1MiB/4.06s] Downloading https://repo.packagist.com/ourcompany/packages.json
[175.4MiB/21.64s] Writing /home/vagrant/.cache/composer/repo/https---repo.packagist.com-ourcompany/packages.json into cache
[155.8MiB/21.68s] Updating dependencies (including require-dev)
[156.0MiB/21.71s] Reading /home/vagrant/.cache/composer/repo/https---repo.packagist.com-ourcompany/provider-myvendor$module-mymodule2-products.json from cache
[156.0MiB/21.71s] Downloading https://repo.packagist.com/ourcompany/p/myvendor/module-mymodule2-products.json
[156.0MiB/23.13s] Writing /home/vagrant/.cache/composer/repo/https---repo.packagist.com-ourcompany/provider-myvendor$module-mymodule2-products.json into cache
................
................
[915.2MiB/46.60s] Resolving dependencies through SAT
The set of these packages is always the same, composer cache size is 2048MiB and increasing it didn't help.
Thanks for help.
Related
The dependency resolution of my Poetry environments frequently takes >20min. My personal best is 6hrs!!! I'm clearly doing something wrong. Running poetry lock -vvv I notice various versions of sdist get downloaded this takes several seconds each time. Additionally, I see to following messages a repeating
or
It seems this is where the resolution takes the longest. I am using a private PyPi server as my secondary
[[tool.poetry.source]]
name = "private_pypi"
url = "https://pypi.private_pypi.com.au/simple"
secondary = true
[[tool.poetry.source]]
name = "pypi-public"
url = "https://pypi.org/simple/"
I also see a message earlier in the logs Private PyPi: Response url ... differs from request url ... not sure if this is related.
There is an extensive thread in the Poetry Git issue 2094 that seems to indicate many of the resolution woes are out of Poetry's hands. Not sure if this is the case for me.
I'm simply looking for some next steps to try and speed things up.
Will nailing down versions in my pyproject.toml (ie: using == not ^ or >=) help? Is there something immediately obvious that I'm doing wrong? How many goats should I sacrifice?
First run:
poetry cache clear pypi --all
Then run
poetry lock
Using CircleCI - version: 2.1 - for continuous deployment where caching installed dependencies. Based on save_cache documentation:
Generates and stores a cache of a file or directory of files such as dependencies or source code in our object storage. Later jobs can restore this cache.
Current scenario:
See the simplified caching step below in .circleci/config.yml file:
steps:
- node/with-cache:
steps:
- checkout
- run: npm install
- save_cache:
key: dependencies
paths: node_modules
The problem is coming once adding new package to the project thus package.json file is changing. In the same time CircleCI shows the message for Saving Cache step:
Skipping cache generation, cache already exists for key: dependenciesFound one created at 2020-05-23 19:29:29 +0000 UTC
Then once restoring the cache obviously does not find the newly added package in the build step:
./src/index.tsxCannot find module: 'package-name'. Make sure this package is installed.
Questions:
Is there any way to check package.json changes in the pipeline? Ideally I would install the dependencies only in those cases, so the cache can be purged and updated.
Maybe I did not see something in the documentation. Any help is appreciated, thank you!
The problem is the cache key you used is "dependencies", a plain string. This key never changes, so you will always use the same exact cache.
You need to use a cache key that changes, preferably based on package.lock. Please read the section of cache keys in the CircleCI Docs for more information: https://circleci.com/docs/2.0/caching/#using-keys-and-templates
I'm trying to setup an Artifactory PHP Composer repository.
In Artifactory repository configuration, we got to set 2 parameters:
- URL
- Registery URL
For Drupal 8 with packages.drupal.org/8 as source of drupal packages, I tried with those values:
- URL: https://git.drupal.org/
- Registery URL: https://packages.drupal.org/8
Error I've got:
Your requirements could not be resolved to an installable set of
packages.
Problem 1
- The requested package drupal/core could not be found in any version, there may be a typo in the package name. Problem 2
- The requested package drupal/tvi could not be found in any version, there may be a typo in the package name. Problem 3
- The requested package composer/installers could not be found in any version, there may be a typo in the package name.
I followed jfrog.com/confluence/display/RTF/PHP+Composer+Repositories
Is there anyone who successfully configured Artifactory PHP Composer for Drupal 8 and which parameters you used?
What I notice, there is little difference between the 2 composer repository
For packagist.org/packages.json
notify "/downloads/%package%"
notify-batch "/downloads/"
providers-url "/p/%package%$%hash%.json"
search "/search.json?q=%query%&type=%type%"
For packages.drupal.org/8/packages.json
notify-batch "/8/downloads"
providers-url "/8/%package%$%hash%.json"
search "/8/search.json?s=%query%"
In Artifactory PHP Composer repo, is there any Hard Coding to report?
Ticket open and visible at https://www.jfrog.com/jira/browse/RTFACT-15188
Thanks
On an existing project with a long list of packages and various feature branches where new dependencies are being added I want to mitigate and minimize merge conflicts by adding dependencies in alphabetical order.
To get this cleaned up, though, I'd like to be able to run the --sort-packages functionality on its own -- without adding or updating anything -- as just a single commit that cleans up the existing packages, and then add "sort-packages" : "true" to the "config" section of the composer.json file to ensure all new packages are added in alphabetical order going forward.
Is it possible to sort the packages listed in a messy composer.json file using composer's --sort-packages option on the CLI without actually adding or updating any dependencies?
The only workaround I've found so far is to run composer update some/package --sort-packages against a package that you're sure wont update because it is already at the latest version. This is not ideal.
I know I'm (really) late! But perhaps my answer can help others in the future ;-)
If you run
composer config sort-packages true
this will add the following in your composer.json file:
"config": {
"sort-packages": true
},
The next time you do a composer require (or update) it will automatically sort the whole list.
You can "re-require" a package you've already required. In my case, it's Symfony 3.4, so I did:
composer require symfony/symfony:3.4.*
If you don't have "sort-packages": true in your composer.json, you can do:
composer require --sort-packages symfony/symfony:3.4.*
From what I can tell, only the require command has the option for sorting packages so it seems you need to require a package for sorting to be applied.
It is since 1.0.0-alpha10 - released 2015-04-14 - that composer require has the --sort-packages option:
Added --sort-packages option to require command for sorting dependencies
This is not yet the composer.json#/config/sort-packages which is since 1.0.0-beta1 - released 2016-03-03, roughly a year later:
Added sort-packages config option to force sorting of the requirements when using the require command
It stems from pull-request #3549 and there is also a bit of backstory as well as more options discussed in Normalizing composer.json by
Andreas Möller (localheinz), Jan 2018.
First of all, +1 for using "sort-packages": true, for all the reasons you describe.
It is a bad idea to edit composer.lock directly, but I do not think the same applies to composer.json. I would edit the files in Vim, select everything inside the "require" and "require-dev" sections (one at a time) and :sort. Plus some fiddling to make sure that every line except the last has a comma.
Using puphpet a typical config.yaml php section might look like this:
php:
install: '1'
settings:
version: '70'
modules:
php:
- ioncube-loader
I have lots of other PHP modules installed but its ioncube I'm having issues with.
Up until just earlier, this was provisioning with PHP 7.0.4, which seemed to have a yum package for Ioncube loader.
I just had to rerun vagrant provision and now all of a sudden I appear to be on PHP 7.0.5.
Not much of a bother normally, except now I get issues with dependencies for ioncube (ioncube loader is crucial to run some encrypted 3rd party code I need).
My question is: Is it possible to lock php down to 7.0.4?
version: '704' doesn't work.
Also just to check I've understood the error message correctly:
Error: Package: php-ioncube-loader-5.1.2-1.el6.remi.5.4.x86_64 (remi)
Requires: php(zend-abi) = 20100525-x86-64
Installed: php-common-7.0.5-1.el6.remi.x86_64 (#remi-php70)
php(api) = 20151012-64
Available: php-common-5.4.45-5.el6.remi.x86_64 (remi)
php(api) = 20100412-x86-64
Available: php-common-5.4.45-7.el6.remi.x86_64 (remi)
php(api) = 20100412-x86-64
Available: php-common-7.0.4-1.el6.remi.x86_64 (remi-php70)
php(api) = 20151012-64
Available: php55u-common-5.5.33-1.ius.centos6.x86_64 (ius)
php(api) = 20121113-64
Available: php56u-common-5.6.19-1.ius.centos6.x86_64 (ius)
php(api) = 20131106-64
Available: php70u-common-7.0.4-1.ius.centos6.x86_64 (ius)
php(zend-abi) = 20151012-64
Other suggestions welcome, I just need ioncube loader working on this 1 project, am I right here? It looks like its available for every version of php except the one that's magically appeared on mine since the last provision. It's totally possible I'm barking up the wrong tree....
Author of PuPHPet here.
Unfortunately subversions are not supported (7.0.x), only major versions (7.0).
This is due to the absolutely insane way each distro and even each PHP version is so different to the other in regards to INI locations and module directories. It's ridiculous.
What could easily be added is support for ensure field to the YAML file that you could pass your specific version to. Mind opening a ticket on my github tracker to enable this?
hum
version: '704' doesn't work
should be
settings:
version: '7.0.4'
ioncube loader is not yet compatible with PHP 7, whatever version you install (7.0.4 or 7.0.5).
See the Compatibility list
Also see: upstream forum thread