I am trying to reduce the Class A operations
on a gcs bucket which is being configured to store yarn and spark history logs.
This is costing us a lot. I disabled spark logs editing the spark-defaults.conf
file.
for every spark job runs, it is adding yarn logs to the gcs bucket.
can I exclude below properties from yarn-site.xml
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
<description>Enable remote logs aggregation to the default FS.</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>gs://dataproc-temp-europe-north1-89560059592-dqicljms/d0decf20-21fd-4536-bbc4-5a4f829e49bf/yarn-logs</value>
<description>
The remote path, on the default FS, to store logs.
</description>
</property>
Do I need to exclude both or yarn.log-aggregation-enable=false
will work?
enabling them again will also work when we need them