Custom retention for blueprint entities | Voters

Custom retention for blueprint entities

Mor Paz - Port team

Some data models include blueprints such as CI/CD jobs or deployments. These blueprints usually get filled with high number of entities, but the entities in these blueprints become stale and lose relevancy very quickly.

With this feature, users will be able to configure custom retention for specific blueprints, giving them the option to automatically delete old entities once the number of entities for the blueprint passes X entities, or if an entity was updated more than 1 week/month/year ago

September 10, 2023

Tomas Kratky

How could it work when the automatically deleted entities (due to ttl) were previously created via data sources such as k8s, GitHub, AWS, and so on? I guess these data-sources would create them again once the automation deletes them, right?

Dudi Elhadad

Hi Tomas Kratky ,

Do you mean a case where you deleted the entity using an automation? If that's the case, and the entity exists in the third-party system (GitHub, AWS, etc), then the data source will indeed create the entity again in the next resync. It will also delete the entity if it is deleted in the third-party system.

Tomas Kratky

Dudi Elhadad I see, thank you for the answer. Will this feature include some mechanism to handle this situation? I can take archived repositories as an example. When the repository is archived, I only want to keep it in the port for the next two quarters but I guess it'll stay in our GitHub org forever.

Dudi Elhadad

Tomas Kratky: This feature is in initial phases, so we don't have a complete design. But this is important feedback. I assure you we will think about this as part of the feature design 🙏

Gur Shafriri

This can now be done with Automation:

when creating an entity, also add a TTL timer field set to the retention timeframe.
Create an automation that will delete this entity when TTL expired

Dipak Parmar and others Let me know what you think

Glenn Moen

Gur Shafriri This seems like a work around for the feature request. Was that the intention? For my use cases, it would be better to set this at the blueprint level instead of the entity level. If I want to add retention to an existing entity, it would require updating all entities. If I decided to change a retention period, it would require updating all entities.

Gur Shafriri

Glenn Moen yes 100% a workaround - it would be ideal to have it as a built in blueprint feature. 
I also didn't consider bulk changes as you mentioned - these can also be done with automation/migration but I see how it can become complex. 
Anyway, I wanted to share a way to currently do this until this get's prioritized as a feature in case it helps someone.

Glenn Moen

Gur Shafriri Thanks for the response and clarification.

Mark Tarry

Gur Shafriri I've already taken this approach - with 2 different blueprints so far:

Deployments - where each time we create a related
deployment attempt
entity, we reset a
data_retention
timer property (on the
deployment
entity) to "now + X days"
Pull Requests - where we set a timer property to 12 months after the
pull_request
entity itself is created

The workaround works - and we're able to streamline adding this (and the associated automations) thanks to some Terraform magic. But it's still more work on our part to implement and test each one.

Dipak Parmar

+1 for this, we have excessive of amounts jobs that runs and we need them in port, but after certain days they are not useful, clutters the system. we can keep the in cicd system, not necessary we need them in port, aggregate is okay.