I had some spare time so I decided to check how many extra dependencies we have and maybe try to remove them. The end result of this journey was quite surprising to me: from over 3000 dependencies that we had I could remove over 1500 without breaking anything. Removing extra dependencies has positive effect on many levels (build time, IDE development, complexity, ...) so if you have a big Maven build somewhere you might want to do the same. I wrote some scripts on the way and decided to open source them (https://github.com/siivonen/maven-cleanup). I also decided to write following instructions for anyone who want's to clean up their Maven dependencies.
0. You can get the scripts to your machine with the following command
git clone git://github.com/siivonen/maven-cleanup.git
1. Make sure that the build works on your machine
For starters you need a green baseline. Figure out a build command that will include all the necessary tests etc in our project. We need a Maven command that is used to determine if everything is OK after a dependency removal. You save yourself time when you select a command that runs all the tests and activates all the relevant profiles. The command you choose should have 'install' in it. The default selection is 'mvn clean install'
Once you know which Maven command you want to use, run a full build using this command on your machine. This build needs to succeed so make all the required tasks to get the build green. In big Maven builds this can be hard sometimes. It is important that every Maven project is built successfully locally before you continue. This way your local repository will have all the needed artifacts and we know that at least before any dependency removals the build was green.
2. Systematically remove dependencies not needed directly
This is rather time consuming and mechanical task so I wrote a script for this. The script takes a root pom.xml file and Maven command as parameters. What it does is following:
- Find all sub modules of given pom.xml file
- For every pom.xml found remove one dependency at a time and build the project with Maven given command
- If the build is successful leave the dependency out
./remove-extra-dependencies.rb pom.xml 'mvn clean install -N'
The script took 14 hours to loop all our 3000+ dependencies in 190 pom.xml files. After the script execution there were 1600 dependency removals to commit. Unfortunately committing those would have resulted in build failure. Removing a dependency from a project will cause build failure on dependent projects that used to get the removed dependency transitively. That's why we often need step 3.
3. Add the missing dependencies
The missing dependencies are something that a project used to get transitively but not anymore. Scripting this task seemed too complex so I decided to do this manually. The process was pretty much the following:
- Run full build command (the one you ran in step 1)
- If the build fails for missing classes:
- search the missing dependency from your local repository
- grep my.missing.Class ~/.m2/repository -r --include *.jar
- add the dependency to failing Maven module (if the test compilation/execution is failing the test scope is enough for that dependency)
- resume the build
- mvn clean install -rf :failing-module
- Repeat this until you get green build
4. Remove extra dependencyManagement entries
After removing a lot of dependencies you probably have several extra dependency management entries. Removing them is again rather mechanical so I scripted that. The script takes a pom.xml path as parameter and:
- Searches all dependencies of the given module and all it's sub modules
- Loop through all the dependency management entries and remove the entries that are not found in the group of all dependencies
./remove-extra-dependency-management.rb pom.xml
This script doesn't need to run any Maven builds so the execution time is rather fast. You might get a build failure after running this script also. The script removes all the dependency management entries that are not referenced directly. You can use dependency management to control transitive dependencies of 3rd party libraries that you are not directly referencing. Also your build might rely on a dependency management exclusions that are now removed. If you have these build problems you can solve them case by case in the similar fashion as in step 3. Once you get a green build after this step you are done!
sounds cool and ~~ handy sort of
ReplyDeleteYou are a hero. I happen know this particular project well enough to say there's a lot of mess to be cleaned up. Next write a script that will re-organize all the code into the optimal structure :)
ReplyDeleteGreat work!! What about using mvn dependency:analyze in step 2?
ReplyDeleteThat could work in some projects. To my understanding it only reports dependencies referenced directly from code so it equals to running 'mvn test-compile' successfully. In our project that would have resulted in a lot of false positives (all the reflection and dependency injection cases, cases that are caught only in integration or browser tests, ...) and therefore a long step 3.
DeleteThanks Samuli, I'm giving this a go now. A couple of things to note:
ReplyDelete1. I'm running on Windows (not by choice), and if anyone else wants to give it a go on Windows it is pretty easy to get the script to run by changing the /dev/null redirection
2. Sometimes parent modules (or assemblies) in Maven will contain dependencies for the sake of simplifying their descendants - this is the case in my current project. If we run the script as-is, it will remove all dependencies from the parent (structural) pom, thereby breaking most if not all children.
For example, we have a library of handy unit testing utilities, and this is included at the top level to avoid repeating this in each project. If the cleanup script removes this it breaks all child projects (and the analysis therein will not work).
My solution to this was to add some code to skip the dependency removal step if the pom contains pom, and this works for our assembly projects as well. I don't really know Ruby, so you could probably do this more elegantly than I can, but I'll share my changes back if you wish.
This sounds like OK solution and it's simple. Another solution that comes to my mind is to run the script separately to parents and children with different Maven command. Parent builds will be recursive and child builds will not. This is more complex and time consuming but results in removing the unused dependencies from the parent projects also.
DeleteIf you have any improvements to the scripts just ship it in. To me content is more important than code.
how do i change /dev/null redirection? Can you be more specific since I know nothing about Ruby? thanks
DeleteI don't have Windows to try this but maybe this helps you: http://stackoverflow.com/questions/313111/dev-null-in-windows. You can also try to install Cygwin on your Windows and run the scripts in Cygwin console without any modifications.
DeleteChange Line 53 to if system("cd #{File.dirname(pom)} && #{ARGV[1]} >nul 2> nul")
DeleteThis comment has been removed by the author.
ReplyDeleteI was wondering, if the manual step 3 is needed, if you would execute the command "non-recursive" in "inverse order" of dependencies in a multi-module build.
ReplyDeleteSo you start by your "leave module(s)", without any dependencies to other modules, and go up the tree and do module by module.
To automate the whole thing then, there is probably a way to "detect inter module dependencies" automatically, and sort the cleaning accordingly.
Regards,
Carsten
This sounds like a good idea. You could probably get a good build order by running 'mvn validate' and reading the 'Reactor Build Order:' from the output.
ReplyDeleteIt is a great piece of work. Saved me a lot of hassle and time. Thank you very much for sharing it.
ReplyDeletePlease don't mind me asking, is there a way to detect deployment time errors? The build works fine after cleaning but I see deployment time errors :-(
Thanks! Deployment time dependencies are harder. Haven't figured out a good solution to that. Trying to cover all the execution paths of live application in the 'mvn test' phase helps (you can use the method described in this blog). The "tests" can just trigger production code and check that no errors are happening.
DeleteIf you have good browser test set you can maybe use that. Remove one dependency at a time, run the browser test set and check if everything is working and log is clean.
Spring uses xml configurations. It would be useful to scan those files too for classes that are used.
ReplyDeleteI wonder; what if the spring xml is generated into a java class with annotations, and compile this too.
شركة مكافحة حشرات بخميس مشيط
ReplyDeleteشركة مكافحة حشرات ببيشة
شركة مكافحة حشرات بحائل
شركة مكافحة حشرات بنجران
ReplyDeleteشركة مكافحة حشرات بجازان
شركة مكافحة حشرات بابها
شركة مكافحة حشرات بالقصيم
ReplyDeleteشركة مكافحة حشرات بينبع
شركة مكافحة حشرات بالخرج
اهلاً ومرحباً بكم عملائنا الكرام نحن نقدم خدمان منزلية مميزة وذات ضمان وجودة عالية جدا عليكم بالتواصل معنا الأن وسوف نلبي طلباتكم بكافة تفاصيلها عن طريق بعض الروابط الخاص بالموقع الخاص بالشركة :.
ReplyDeleteشركة عزل اسطح بابها
شركة نقل عفش بابها
شركة عزل خزانات بابها
شركة تنظيف مجالس بابها
شركة تنظيف شقق بابها
شركة مكافحة النمل الأبيض بابها
شركة ترميم منازل بابها
شركة عزل اسطح بابها
خدمات منزلية بأرخص الأسعار عالية الدقة قمة التميز في إختيار ماكينات التنظيف أيادي عاملة خبرة منذ سنوات كافة عوامل النجاح تتوفر في شركة التميز الجنوبي يمكنكم التتبع والتواصل معنا عبر الروابط التالية :.شركة مكافحة حشرات بابها
ReplyDeleteشركة تنظيف منازل بابها
شركة تنظيف فلل بابها
شركة تنظيف خزانات بابها
شركة تنظيف بابها
شركة من الشركات الرائدة في الخدمات المنزلية شركة التميز الجنوبي تتميز بوجود عمال متميزة وعلى أعلى دقة ممكن شركة التميز الجنوبي أسعار بدون منافسة تواصل معنا الأن عبر الروابط التالية وسوف نلبي طلباتكم في اسرع وقت ممكن :.
ReplyDeleteشركة مكافحة نمل أبيض بخميس مشيط
شركة مكافحة حشرات بخميس مشيط
شركة تنظيف بخميس مشيط
شركة تنظيف خزانات بخميس مشيط
شركة تنظيف شقق بخميس مشيط
شركة تنظيف فلل بخميس مشيط
شركة تنظيف مجالس بخميس مشيط
شركة عزل خزانات بخميس مشيط
ReplyDeleteشكة عزل اسطح بابها
شركة عزل اسطح بخميس مشيط
شركة كشف تسربات المياه بخميس مشيط
شركة كشف تسربات المياه بابها
شركة مكافحة حشرات بجازان
ReplyDeleteشركة مكافحة الحمام بجازان
شركة رش مبيدات بجازان
شركة رش دفان بجازان
شركة كشف تسربات المياه بجازان
شركة تركيب طارد حمام بجازان
Hi there! I'm at work browsing your blog from my new apple iphone! Just wanted to say I love reading your blog and look forward to all your posts! Carry on the excellent work!
ReplyDeleteCleanliness is from faith andit is one of the necessary things that must be taken care of. Life without hygiene becomes full of epidemics and diseases, and from here all housewives care about cleanliness and use a cleaning company to help them in all house cleaning operations
ReplyDeleteشركة تنظيف ببريدة
شركة تنظيف مجالس ببريدة
شركة مكافحة حشرات ببريدة
شركة نقل عفش ببريدة
شركة كشف تسربات المياه ببريدة
Nice Blog.Thank you for sharing this valuable content.
ReplyDeleteDevOps Training
DevOps Online Training
Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
ReplyDeletedevops online training
best devops online training
top devops online training
Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
ReplyDeletesplunk online training
Thanks for sharing such a good blog.
ReplyDeleteDevOps Training
DevOps Online Training
Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
ReplyDeletedevops online training
best devops online training
top devops online training
بشركه تسليك مجارى بالرياض
ReplyDeleteبشركه رش مبيدات بالرياض
شركه نقل عفش بالرياض
شركه رش مبيدات حشريه بالرياض
شركه نقل اثاث بالرياض
شركه عزل حراري بالرياض
شركه عزل مائي بالرياض
شركه كشف تسربات المياه بالرياض
cami avizesi - cami avizeleri - avize cami - no deposit bonus forex 2021 - takipçi satın al - takipçi satın al - takipçi satın al - takipcialdim.com/tiktok-takipci-satin-al/ - instagram beğeni satın al - instagram beğeni satın al - btcturk - tiktok izlenme satın al - sms onay - youtube izlenme satın al - no deposit bonus forex 2021 - tiktok jeton hilesi - tiktok beğeni satın al - binance - takipçi satın al - uc satın al - sms onay - sms onay - tiktok takipçi satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - instagram beğeni satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - takipcialdim.com/instagram-begeni-satin-al/ - tiktok takipçi satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - instagram beğeni satın al - perde modelleri - instagram takipçi satın al - takipçi satın al - instagram takipçi satın al - betboo
ReplyDeleteinstagram takipçi satın al
ReplyDeleteaşk kitapları
tiktok takipçi satın al
instagram beğeni satın al
youtube abone satın al
twitter takipçi satın al
tiktok beğeni satın al
tiktok izlenme satın al
twitter takipçi satın al
tiktok takipçi satın al
youtube abone satın al
tiktok beğeni satın al
instagram beğeni satın al
trend topic satın al
trend topic satın al
youtube abone satın al
instagram takipçi satın al
beğeni satın al
tiktok izlenme satın al
sms onay
youtube izlenme satın al
tiktok beğeni satın al
sms onay
sms onay
perde modelleri
instagram takipçi satın al
takipçi satın al
tiktok jeton hilesi
instagram takipçi satın al
pubg uc satın al
sultanbet
marsbahis
betboo
betboo
betboo
instagram takipçi satın al
ReplyDeleteucuz takipçi
takipçi satın al
https://takipcikenti.com
https://ucsatinal.org
instagram takipçi satın al
https://perdemodelleri.org
https://yazanadam.com
instagram takipçi satın al
balon perdeler
petek üstü perde
mutfak tül modelleri
kısa perde modelleri
fon perde modelleri
tül perde modelleri
https://atakanmedya.com
https://fatihmedya.com
https://smmpaketleri.com
https://takipcialdim.com
https://yazanadam.com
yasaklı sitelere giriş
aşk kitapları
yabancı şarkılar
sigorta sorgula
https://cozumlec.com
word indir ücretsiz
tiktok jeton hilesi
rastgele görüntülü sohbet
erkek spor ayakkabı
fitness moves
gym workouts
https://marsbahiscasino.org
http://4mcafee.com
http://paydayloansonlineare.com
marsbahis
ReplyDeletebetboo
sultanbet
marsbahis
betboo
sultanbet
bet88 1xbet korean | Legalbet
ReplyDeletebet88 1xbet korean. bet88 1xbet korean. bet88 1xbet korean. bet88 1xbet korean. bet88 1xbet korean. bet88 1xbet korean. bet88 1xbet korean. bet88 1xbet 1xbet promo code korean.
thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website devops Online Training
ReplyDeletebest devops Online Training
top devops Online Training
instagram beğeni satın al
ReplyDeleteyurtdışı kargo
seo fiyatları
saç ekimi
dedektör
fantazi iç giyim
sosyal medya yönetimi
farmasi üyelik
mobil ödeme bozdurma
bitcoin nasıl alınır
ReplyDeletetiktok jeton hilesi
youtube abone satın al
gate io güvenilir mi
referans kimliği nedir
tiktok takipçi satın al
bitcoin nasıl alınır
mobil ödeme bozdurma
mobil ödeme bozdurma
mmorpg oyunlar
ReplyDeleteinstagram takipçi satin al
tiktok jeton hilesi
tiktok jeton hilesi
antalya saç ekimi
referans kimliği nedir
İNSTAGRAM TAKİPÇİ SATIN AL
metin2 pvp serverlar
INSTAGRAM TAKİPCİ
smm panel
ReplyDeleteSMM PANEL
İS İLANLARİ BLOG
instagram takipçi satın al
hirdavatciburada.com
beyazesyateknikservisi.com.tr
Servis
tiktok jeton hilesi
kadıköy bosch klima servisi
ReplyDeletebeykoz arçelik klima servisi
pendik mitsubishi klima servisi
maltepe arçelik klima servisi
kadıköy arçelik klima servisi
kartal samsung klima servisi
ümraniye samsung klima servisi
kartal mitsubishi klima servisi
ümraniye mitsubishi klima servisi