Zu Hauptinhalten wechseln

FV Decipher Unterstützung

Alle Themen, Ressourcen für FV Decipher benötigt.

 
decipher

Advanced Unix Commands for Programmers

  Requires Decipher Cloud

Some things to note before you begin:

  • Almost all Unix commands are case sensitive
  • After almost every command, you can add --help to find out more information (e.g. grep --help)
  • The asterisk (*) is used to match any and all characters
    (e.g. */survey.xml matches the survey.xml file in every directory from your current position)
  • The period . means the current directory
  • The double period .. is the previous directory
  • Chain commands together with the pipe | symbol (e.g. ls | grep "proj14" | sort)
  • Output the results of any command with the greater-than symbol >
    (e.g. ls > directory_contents.txt # creates/overwrites the directory_contents.txt file)
    (e.g. ls >> directory_contents.txt # appends to the directory_contents.txt file)
  • Use CTRL + C to cancel a current running command. (Press repeatedly if needed)

1:  Grep (i.e. Searching for a String in a File)

The unix command `grep` is a powerful tool that can be used to search for strings within files.

For example, if we wanted to find out what kind of question "Q1" is in our survey.xml file, we could run the following command from our project's directory:

Input

grep 'label="Q1"' survey.xml

Output

<radio label="Q1">

Since we found an exact match in the survey.xml file, we can see that Q1 is a <radio> question.

What if we wanted to see what type of question "Q1" is for every project in our company directory? We can use the asterisk to match all directories containing a survey.xml from our company's directory:

Input

grep 'label="Q1"' */survey.xml

Output

proj13040b/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13040/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13041/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13043/survey.xml:<textarea label="Q1" optional="0">
proj13050/survey.xml:<text label="Q1" optional="0" style="exclusiveCol">
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13055/survey.xml:<checkbox label="Q1" atleast="1">
proj13056/survey.xml:<checkbox label="Q1" atleast="1">
proj13059/survey.xml:<select label="Q1" optional="0">

We can see that "Q1" reflects a variety of question types in several of our projects.

In that same example, if we only wanted to view the directories where "Q1" was used as a label, we can add -l to our command to print only names of files that contain a match.

Input

grep -l 'label="Q1"' */survey.xml

Output

proj13040b/survey.xml
proj13040/survey.xml
proj13041/survey.xml
proj13042/survey.xml
proj13043/survey.xml
proj13050/survey.xml
proj13052/survey.xml
proj13055/survey.xml
proj13056/survey.xml
proj13059/survey.xml

Only the files that contain an exact match were returned. Perfect.

To perform a case-insensitive search, in case "Q1" was written as "q1", we can add -i to our command:

Input

grep -i 'label="Q1"' */survey.xml

Output

proj120200/survey.xml:<radio id="sJtiq" optional="0" label="q1" builder:autoGrouping="1">
proj120701/survey.xml:<number id="UZLDT" label="q1" optional="0" size="10">
proj120800/survey.xml:<number id="UZLDT" label="q1" optional="0" size="10">
proj13040b/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13040/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13041/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13043/survey.xml:<textarea label="Q1" optional="0">
proj13050/survey.xml:<text label="Q1" optional="0" style="exclusiveCol">
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13055/survey.xml:<checkbox label="Q1" atleast="1">
proj13056/survey.xml:<checkbox label="Q1" atleast="1">
proj13059/survey.xml:<select label="Q1" optional="0">

With the results above, we can see that projects with a lable of "q1" and "Q1" were now shown.

We can use -A# and -B# to also grab a number of lines after or before the line that matches our search. The command below will search for all radio questions with a label of "Q1" in each survey.xml for every directory in our current directory.

Input

grep -A2 '<radio label="Q1"' */survey.xml

Output

proj120500/survey.xml: <radio label="Q1" shuffle="rows" id="DUAP7">
proj120500/survey.xml-    <title id="sg7J9">Overall how familiar are you with the 4:30 to 7:00 a.m. local newscasts on the following news stations?</title>
proj120500/survey.xml-    <row label="r1" id="yPuQF">Channel 3 - KSNV</row>
--
proj130500/survey.xml: <radio label="Q1" id="I0NS5">
proj130500/survey.xml-    <title id="AfWT7">How many hours a week do you spend playing computer or video games?</title>
proj130500/survey.xml-    <row label="r1" id="fHsBI">Less than 5 hours per week</row>
--
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13042/survey.xml-  <title>
proj13042/survey.xml-    How often do you do the following?
--
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13052/survey.xml-   <title>Do you <u>watch</u>, or record to view at a later time, any daytime dramas or soap operas that air on television, including cable or satellite?</title>
proj13052/survey.xml-   <comment>Select one<br/><br/></comment>
--

The command grabbed the line containing our match plus 2 more lines following the matched line.

If we had the following email list:

horse@coconuts.com
knight@ni.com
swallow[at]unknownlocation.com
sirlancelot@camelot.com
troll@bridge.com

To find the lines that don't contain a valid email address with the "@" symbol, we can add -v to our grep command to select non-matching lines.

Input

grep -v "@" email-list.txt

Output

swallow[at]unknownlocation.com

Learn more:

man grep

2:  Find (i.e. Searching for a File in a Directory)

The unix command `find` will navigate child directories for whatever string you want to find:

To get a feel for what find does, type it in and see what happens..

Input

find

For example, if we wanted to search for all directories containing a /lang directory, use the following command:

Input

find -name lang

Output

./proj14005/temp-final-20140109/lang
./proj14005/lang
./proj14005/temp-spanish/lang
./proj13104/lang
./proj13118a/lang
./proj13108/temp-final-v2/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-v3/lang
./proj13108/lang

Use the find command and pipe the results to grep to achieve a similar effect.

Input

find | grep lang

Output

./proj14005/temp-final-20140109/lang
./proj14005/lang
./proj14005/temp-spanish/lang
./proj13104/lang
./proj13118a/lang
./proj13108/temp-final-v2/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-v3/lang
./proj13108/lang

Similarly, we can add another pipe to sort the results.

Input

find | grep lang | sort

Output

./proj13104/lang
./proj13108/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-v2/lang
./proj13108/temp-final-v3/lang
./proj13118a/lang
./proj14005/lang
./proj14005/temp-final-20140109/lang
./proj14005/temp-spanish/lang

The results have been sorted alphabetically in ascending order.

Now let's find all lang directories that were created in 2014.

Input

find proj14*/ -name lang

Output

proj14005/temp-final-20140109/lang
proj14005/lang
proj14005/temp-spanish/lang

3:  Head & Tail (i.e. Reading a Number of Lines)

Head - output the first part of files

Tail - output the last part of files

To get a feel for what these commands do, choose a file (e.g. file.txt) and running the following commands:

head -2 file.txt
tail -5 file.txt

The head command above will output the first two lines of the file and the tail command will output the last 5 lines of the file. The numbers provided can be any number.

For example, if we had a list of emails (e.g. email-list.txt) to send our campaign to, we could get a soft send list (e.g. soft-send.txt) of 5000 using the following command:

head -5000 email-list.txt > soft-send.txt

This will create/overwrite the soft-send.txt file with the first 5000 lines of our email-list.txt file.

To create a full-send.txt file containing the remaining email addresses to send to, we would then find out how many lines exist in the email-list.txt and use the tail command to get the rest of the list. Assuming there were 50,000 lines, the following command would do:

tail -45000 email-list.txt > full-send.txt

Similarly, a bit more advanced approach would be to calculate the number of lines on-the-fly.

tail -$(expr $(cat email-list.txt | wc -l) - 5000) email-list.txt > full-send.txt

The example above uses 3 different sets of commands to achieve the final result.

cat list.txt | head -n 10001 > list.softSend.txt

Sort

4:  Sort

The `sort` command will, as you might have guessed, sort content for you. Look at the following workflow to get a feel for what sort is capable of.

$ cat list.txt
4
a
3
d
2
a
b
4
7
1
$ sort list.txt
1
2
3
4
4
7
a
a
b
d
$ sort -r list.txt
d
b
a
a
7
4
4
3
2
1
$ cat list.txt | sort | uniq
1
2
3
4
7
a
b
d

Learn more:

man sort

5:  Awk

Awk is a very advanced tool used to process and alter files, one line at a time.

To begin, let's take the following tab-delimited file (list.txt) and process it using awk.

1 2 3 4 5
1 2 3 4 5
1 2 3 p 5
1 2 3 4 5

To iterate through each line and print the entire line:

Input

awk '{print $0}' list.xt

Output

1 2 3 4 5
1 2 3 4 5
1 2 3 p 5
1 2 3 4 5

Nothing magical here, just note that $0 means the entire line. If we only want to print the 3rd column:

Input

awk '{print $3}' list.xt

Output

3
3
3
3

If we wanted to change the 4th column to the number 6:

Input

awk '{$4 = "6"; print $0}' list.txt

Output

1 2 3 6 5
1 2 3 6 5
1 2 3 6 5
1 2 3 6 5

If we wanted to change the 4th column to the number 6 only if the 4th column is the number 4:

Input

awk '{if ($4 == "4") $4 = "6"; print $0}' list.txt

Output

1 2 3 6 5
1 2 3 6 5
1 2 3 p 5
1 2 3 6 5

Learn more:

man awk

6:  Find and Replace with tr

The `tr` command is able to replace or remove characters inputted into it. The number of characters must be the same.

In this example, we replace all spaces with tabs:

cat file.txt | tr ' ' '\t' > out.txt

As shown below, we can also use tr with sets.

$ echo "testing" | tr "a-z" "A-Z"
TESTING
$ echo "tesTing" | tr "[:lower:]" "[:upper:]"
TESTING
$ echo "testing123" | tr "[:alpha:]" "a"
aaaaaaa123

7:  Find & Replace with sed

The command `sed` uses regular expressions to find and replace content within a file. For example, to replace all occurrences of "foo" with "foobar" in file.txt, use the following command:

cat file.txt | sed "s/foo/foobar/g" > out.txt

The "/g" means find and replace to every instance in the file.

We can use sed to quickly add a new column to a file:

sed "s/^/foo\t/g" input.txt > out.txt

We can use sed to add one to the end of the file:

sed "s/$/\tfoo/g" input.txt > output.txt

To find all the IF's or exec's within a survey.xml:

Input

cat survey.xml |  sed -n '/<if cond/,/<\/if>/p' > out.txt or cat survey.xml |  sed -n '/<exec>/,/<\/exec>/p' > out.txt

Output

<if cond="'nolist' in p.markers or 'noclist' in p.markers">
  <comment label="missing" where="survey"><div class='error'>
You are missing information in the URL. Please verify the URL with the original invite.
</div>;
<br/><br/></comment>

  <suspend/>

  <goto target="missing"/>

</if>
<if cond="(1)" randomize="1">
<exec>
_index = ord("$(_letter)") - ord("a") + 1

p._item = "i%s" % _index
p._design = p._designs[0]
p._seller = p._sellers[_index - 1]

p._itemOrder.append(p._item)
</exec>

8:  seq (i.e. Generate a Sequence of Numbers)

The `seq` command can be used to quickly generate a series of numbers. For example, to output the numbers 1 - 100 to a file:

 seq 1 100 > numbers.txt 
1
2
3
4
5
6
7
8
9
10
...
99
100

Similarly, we can print out every odd from 1 - 100:

 seq 1 2 100 > numbers.txt 
1
3
5
7
9
...
99

Learn more:

seq --help

9:  wc (i.e. Word Count)

The `wc` command can be used to count lines, characters, or letters from a file. Given the following file (file.txt):

alpha
omega
beta

We can count the number of lines:

Input

cat file.txt | wc -l

Output

3

We can count the number of characters:

Input

cat file.txt | wc -c

Output

17

We can count the number of words:

Input

cat file.txt | wc -w

Output

3

Learn more:

wc --help

10:  --help & man

To learn more about any Unix command, you can append --help after the command for more information:

grep --help
wc --help
seq --help
cat --help
ls --help

If --help is not available, you can view the man page for any command with the man command (stands for manual):

man grep
man wc
man seq
man cat
man ls

Scripts that were created solely for the Decipher platform may not contain a man page.

11:  Randomly generated list file

Need a list file with random data in it? Run:

here dev/generate-random -n500 source email name list:1,2,3 country:uk,us,eu > somefile.txt

The -n500 determines count of items in the list. The fields source, email and name have a random appropriate value. For other fields, specify fieldname, colon, then a comma-separated list of values.

  • War dieser Artikel hilfreich?