Read CSV file – Python
import pandas as pd import os os.getcwd() tendulkar = pd.read_csv("/dbfs/FileStore/tables/tendulkar.csv", header='infer') tendulkar.shape #os.listdir('dbfs/FileStore/tables/')
Read CSV file – Pyspark
from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext conf = SparkConf().setAppName("CDI Transform") sc = SparkContext.getOrCreate() sqlContext=SQLContext(sc) tendulkar1= (sqlContext .read.format("com.databricks.spark.csv") .options(delimiter=',', header='true', inferschema='true') .load("/FileStore/tables/tendulkar.csv"))
Data frame Shape -Python
Data frame Shape -Pyspark
tendulkar1.count() len(tendulkar1.columns) def dfShape(df): return(df.count(),len(df.columns)) dfShape(tendulkar1)
Data frame columns – Python
Data frame columns – Pyspark
Dtypes – Python
Select columns – Python
Select columns – Pyspark
Filter rows by criteria – Python
b = tendulkar['Runs'] >50 df = tendulkar[b] df.head(10)
Filtering by condition – Pyspark
Display unique contents of a column – Python
tendulkar = pd.read_csv("/dbfs/FileStore/tables/tendulkar.csv", header='infer') tendulkar['Runs'].unique()
Display unique contents of a column – Pyspark
from pyspark.sql.functions import * tendulkar1= (sqlContext .read.format("com.databricks.spark.csv") .options(delimiter=',', header='true', inferschema='true') .load("/FileStore/tables/tendulkar.csv")) tendulkar1.select('Runs').rdd.distinct().collect()
Aggregate mean, max, min – Python
#tendulkar[['Mins','BF','Runs']].groupby('Runs').mean() # Remove rows which have DNB import pandas as pd import os os.getcwd() tendulkar = pd.read_csv("/dbfs/FileStore/tables/tendulkar.csv", header='infer') tendulkar.shape a=tendulkar.Runs !="DNB" tendulkar=tendulkar[a] tendulkar.shape # Remove rows which have TDNB b=tendulkar.Runs !="TDNB" tendulkar=tendulkar[b] tendulkar.shape # Remove the '*' character c= tendulkar.BF != "-" tendulkar=tendulkar[c] tendulkar.Runs= tendulkar.Runs.str.replace(r"[*]","") #tendulkar.shape type(tendulkar['Runs']) tendulkar['Runs']=pd.to_numeric(tendulkar['Runs']) tendulkar['BF']=pd.to_numeric(tendulkar['BF']) df=tendulkar[['Runs','BF','Ground']].groupby('Ground').agg(['mean','min','max']) df.head(10)
Aggregate mean,min max – Pyspark
from pyspark.sql.functions import * tendulkar1= (sqlContext .read.format("com.databricks.spark.csv") .options(delimiter=',', header='true', inferschema='true') .load("/FileStore/tables/tendulkar.csv")) tendulkar1= tendulkar1.where(tendulkar1['Runs'] != 'DNB') print(dfShape(tendulkar1)) tendulkar1= tendulkar1.where(tendulkar1['Runs'] != 'TDNB') print(dfShape(tendulkar1)) tendulkar1 = tendulkar1.withColumn('Runs', regexp_replace('Runs', '[*]', '')) tendulkar1.select('Runs').rdd.distinct().collect() from pyspark.sql import functions as F df=tendulkar1[['Runs','BF','Ground']].groupby(tendulkar1['Ground']).agg(F.mean(tendulkar1['Runs']),F.min(tendulkar1['Runs']),F.max(tendulkar1['Runs'])) df.show()
The Bhagavad Gita is an ancient Hindu text, supposed to have been written more than 5000 years ago. According to Hindu mythology, Krishna recited the verses in the Gita to Arjuna, in the battlefield during the Mahabharata war. However, it is said that many of the sayings in the Gita can probably be traced to earlier Upanishads.
The Gita includes some pretty heady philosophy of the Personal Self (Atman) and the Supreme Consciousness (Brahman). Some of these parts may or may not be relevant in today’s context and body of knowledge. This post will skirt around such philosophical questions and look on some of the more practical sayings of the Gita.
I would like to give a different slant to some of the sayings from the Gita as opposed to the accepted interpretation. This could be a case of the ‘devil citing the Scriptures for his own purpose’. In any case you decide
Gita: You have a right to perform your prescribed duty, but you are not entitled to the fruits of action
The above saying requires us to do our prescribed work without giving thought to the results of the work. However most of us do something, in anticipation of success or a reward of some kind. To a large extent, it is this success/reward, which motivates us to do better. But it appears that the Gita wants us to ignore the reward, but just mindlessly carry on with our work, with total disregard for the outcome.
In my opinion this is not so. If people are not driven by the thought of success, people will simply do things mechanically. I think is important that we always do, what we need to do, to the best of our abilities. While we can be mindful of the taste of success, we should not be so carried away by success that we under-perform. Also on the other hand after having tasted success we should neither become giddy-headed with success nor be broken hearted by failure. We need to be able to face both success and failure with equanimity. So in essence, while we can be driven by success, to perform better, we should not let this come in the way of our performance as either anxiety or over-confidence.
The Gita enjoins us to perform ‘nishkama karma’ or desireless -action. In fact the Gita goes on to say that ‘desire leads to anger, anger leads delusion, from delusion loss of memory and finally loss of spiritual intelligence’. In other words desire is the root of all evil.
This is again a very sticky point. For this we need to understand what desire really is and when it is bad? Some people take the route that all desire is bad. They carry it to the extreme where and claim that one has to be content with whatever life throws one’s way.
My interpretation is that there is a fine line between desire and ambition. To me, it absolutely fine, if you want to become a millionaire, get the snazziest car or desire for a really great house as long as you work honestly towards acquiring it. Desire is only bad when we try to acquire the object of desire through illegal, wrongful methods. However as the Gita says we should not get attached to these worldly belongings.
So after you become a millionaire, if you start getting attached to the millionaire lifestyle then you are in for some deep trouble. So go after what you want, but do not become dependent on these worldly acquisitions. Do not be under the impression, that Gita forbids ambition of any sort.
A true yogi treats success and failure, praise and criticism, good and bad alike.
This saying has the danger of being mis-interpreted that we should neither react to success or failure nor to good or bad things that happen to us in our life. In reality what we need to do, is not get proud because of success, nor be dejected because of failure. If we face problems in our lives we should not face them stoically all the while imagining ourselves to be a true yogi or the ‘biblical Job’. Rather we have to stand up to our problems and take positive action.
So those are my interpretation of some of the more common sayings in the Bhagavad Gita.
If you wonder about the transience of butterflies, whose average life span is about month, here is news for you. Our life span of 70-80 years, is also fleeting and transitory, in the face of the age of the universe. We are were now, gone then!
We live only once in this corporeal body in this planet earth. We are then gone forever.
Under these circumstances it is important to be aware of the following
There is no heaven or hell, no swarga or naraga. There is no afterlife, there is no rebirth. Also there is neither a soul nor is there an atman.
This life, this existence, is all that we have.
Given this fleeting existence, in this lonely planet, that is lost in the wilderness of space, does it make sense to fret, fume, worry, be anxious etc.? It really does not make any sense at all. The suffering, pain, anxieties, worries don’t mean anything in the long run. So the next question is why worry, why be anxious etc.?
Unfortunately it is not easy to break out of this cycle of worry, anxiety, anger and other negative emotions as we have been programmed to behave in this way. These emotions that we experience are the result of centuries of conditioning in our mental makeup. This conditioning forces to react in a particular ways.
When somebody hurts our ego we get angry. When we are humiliated we feel hurt. When we imagine some outcome that is contrary to what we expect, we start to be anxious and begin to worry. We can’t get away from this cycle. However it is good to temper all your reactions with the thought, that your emotions, worries and anger are irrelevant in the larger scheme of things. It may not do away with the ill feeling, but it is still worth a try.
The centuries of mental conditioning will cause the neurons to fire is specific ways, and we can’t make the neurons to ‘unfire’. We can’t swim against this tide of emotions which are reactions to events.
While there is no soul or atman, these may be useful mental constructs to some in helping them to stick to values. The soul is supposed to be ennobled by good deeds and defiled by bad ones. The atma is above buddhi(intellect), manas (mind) and the indriyas( senses and is synonymous with the Self. Supposedly knowledge of the Self will result in bliss and peace.
But the soul or atman don’t really exist. If one needs to take recourse to these concepts to maintain a value driven life it is fine. But we should be aware that these are just concepts.
I was reminded of this John Lennon song, Imagine
Imagine there is no heaven,
It is easy if you try
No hell below us
Above us, only sky
So in essence, in this short span of existence we should live sensibly. We need to look at life in the correct perspective and not get embroiled in imaginary concepts like swarg-narag, heaven-hall, soul-atman.
While we cannot avoid some of the emotions we must be cognizant of the fact that all good and bad experiences must pass and we will leave this corporeal frame forever.
Our life in this lonely planet may be short (~70 years) in comparison to the life of the universe (13.6 billion years). Yet we need to move ahead in this earthly stint of ours. Here are some general tips to move ahead in life.
Become aware of who you are: If you want to move ahead in life the first step that you need to take is to know yourself. You need to understand who you are, where you are in life and where you want to go. Without complete awareness of oneself it is futile to pursue any action for progress.
Introspect and identify your strengths and weaknesses. We can only understand ourselves, if we inquire into our very own nature. We need to fully understand where we are currently in life versus where we would like to reach in life. This help us know the areas we need to focus on and the areas we need to improve upon.
Talent: To move ahead we need an adequate amount of talent. Talent in people occurs in varying degrees. But it is up to each individual to improve on his talent. Even mediocre people can improve their skill and talent in any field they choose, as long as there is ‘deliberate, focused’ practice. While some people are born with talent there are many who acquire the necessary talent through years of perseverance, enthusiasm and dedication. So lack of talent can never be an excuse for not getting to where you want in life. Finally it is attitude, persistence and practice that makes one an expert with all the necessary talent.
Hard work: Perseverance and persistence are the pillars of success in life. One cannot state the importance of this enough. Behind the fame and success of reputed people are years and years of hard work, unswerving dedication and tenacity to not let go. Even successful musicians, sportsmen, scientist have to keep constantly honing their skills to move ahead in their field. Remember the world is full of talented people and there is a constant necessity to push ahead in life besides improving on your skill.
Street Smarts: This skill in a person is often ignored as against other qualities like persistence, talent and hard work. But in the long run street smarts or ‘savvy’ is clearly what distinguishes winners from the also-rans. Savvy is knowing which option amongst the many are the one that count. Smarts is being able to strategize and make appropriate plans for success. Street Smarts in reality, is making the most of the talent and skill that one already has against the competition. It is knowing the difference on when to strike and when to stay put.
Street smarts requires dollops of common sense. There are many situations where a smarter player is victorious over a more talented player simply because of his cleverness in applying what he knows. Life is a game where you skill counts as much as your ability to outwit odds stacked against you.
Hence to move ahead in life we need all the work – Self-awareness, talent, hard-work and finally street-smarts.